Abstract
Cross-modal transfer learning is used to improve multi-modal classification models (e.g., for human activity recognition in human-robot collaboration). However, existing methods require paired sensor data at both training and inference, limiting deployment in resource-constrained environments where full sensor suites are not economically and technically usable. To address this, we propose Decoupled Cross-Attention Transfer (D-CAT), a framework that aligns modality-specific representations without requiring joint sensor modality during inference. Our approach combines a self-attention module for feature extraction with a novel cross-attention alignment loss, which enforces the alignment of sensors' feature spaces without requiring the coupling of the classification pipelines of both modalities. We evaluate D-CAT on three multi-modal human activity datasets (IMU, video, and audio) under both in-distribution and out-of-distribution scenarios, comparing against uni-modal models. Results show that in in-distribution scenarios, transferring from high-performing modalities (e.g., video to IMU) yields up to 10% F1-score gains over uni-modal training. In out-of-distribution scenarios, even weaker source modalities (e.g., IMU to video) improve target performance, as long as the target model isn't overfitted on the training data. By enabling single-sensor inference with cross-modal knowledge, D-CAT reduces hardware redundancy for perception systems while maintaining accuracy, which is critical for cost-sensitive or adaptive deployments (e.g., assistive robots in homes with variable sensor availability). Code is available at this https URL.
Abstract (translated)
跨模态迁移学习被用于提升多模态分类模型（例如在人机协作中的人类活动识别）。然而，现有的方法需要在训练和推理阶段都有配对的传感器数据，这限制了其在无法经济和技术上支持全套传感器设备的资源受限环境中的部署。为解决这一问题，我们提出了解耦跨注意力迁移（Decoupled Cross-Attention Transfer, D-CAT），这是一种框架，在不需联合传感器模态的情况下，就能对特定于模式的表示进行对齐。我们的方法结合了自注意模块用于特征提取，并引入了一种新颖的跨注意对准损失，该损失强制要求在无需两种模式分类管道耦合的情况下，实现传感器特征空间的一致性。 我们在三个多模态人类活动数据集（IMU、视频和音频）上评估了D-CAT，在分布内和分布外场景下都进行了测试，并与单模态模型进行了比较。结果表明，在分布内场景中，从高表现力的模式转移（例如，从视频到IMU），可以比单模态训练高出高达10%的F1得分。在分布外场景中，即使是从较弱的源模式迁移过来（例如，从IMU到视频）也可以提高目标性能，只要目标模型没有过度拟合于训练数据即可。 通过使单一传感器推断具备跨模态知识，D-CAT减少了感知系统中的硬件冗余同时保持了准确性。这对于成本敏感或适应性部署至关重要（例如，在家中具有可变传感器可用性的辅助机器人）。代码可以在该链接处获取：[请在此插入实际的URL]
URL
https://arxiv.org/abs/2509.09747
