Abstract
The core of video-based visible-infrared person re-identification (VVI-ReID) lies in learning sequence-level modal-invariant representations across different modalities. Recent research tends to use modality-shared language prompts generated by CLIP to guide the learning of modal-invariant representations. Despite achieving optimal performance, such methods still face limitations in efficient spatial-temporal modeling, sufficient cross-modal interaction, and explicit modality-level loss guidance. To address these issues, we propose the language-driven sequence-level modal-invariant representation learning (LSMRL) method, which includes spatial-temporal feature learning (STFL) module, semantic diffusion (SD) module and cross-modal interaction (CMI) module. To enable parameter- and computation-efficient spatial-temporal modeling, the STFL module is built upon CLIP with minimal modifications. To achieve sufficient cross-modal interaction and enhance the learning of modal-invariant features, the SD module is proposed to diffuse modality-shared language prompts into visible and infrared features to establish preliminary modal consistency. The CMI module is further developed to leverage bidirectional cross-modal self-attention to eliminate residual modality gaps and refine modal-invariant representations. To explicitly enhance the learning of modal-invariant representations, two modality-level losses are introduced to improve the features' discriminative ability and their generalization to unseen categories. Extensive experiments on large-scale VVI-ReID datasets demonstrate the superiority of LSMRL over AOTA methods.
Abstract (translated)
视频基础的可见光-红外人员再识别（VVI-ReID）的核心在于学习不同模态之间的序列级模态不变表示。最近的研究倾向于使用由CLIP生成的模态共享语言提示来指导模态不变表示的学习。尽管这些方法已经达到了最优性能，但在高效的时空建模、充分的跨模态交互以及明确的模态级别损失引导方面仍然存在局限性。为了解决这些问题，我们提出了一种基于语言驱动的序列级模态不变表征学习（LSMRL）的方法，该方法包括时空特征学习（STFL）模块、语义扩散（SD）模块和跨模态交互（CMI）模块。 为了实现参数高效且计算高效的时空建模，STFL模块是在CLIP的基础上进行少量修改而构建的。为了达到充分的跨模态交互并增强模态不变特征的学习，提出了一个语义扩散模块，将模态共享的语言提示扩散到可见光和红外特征中，以建立初步的模态一致性。此外，通过发展双向跨模态自注意力机制来消除残余模态差距，并细化模态不变表示，进一步改进了CMI模块。 为了明确增强模态不变表示的学习，我们引入了两种模态级损失，以提高特征的判别能力和其对未见类别的泛化能力。在大规模VVI-ReID数据集上的广泛实验表明，LSMRL方法优于现有的AOTA（Attention-based One-to-Many Alignment）方法。
URL
https://arxiv.org/abs/2601.12062
