Abstract
In the domain of multimodal intent recognition (MIR), the objective is to recognize human intent by integrating a variety of modalities, such as language text, body gestures, and tones. However, existing approaches face difficulties adequately capturing the intrinsic connections between the modalities and overlooking the corresponding semantic representations of intent. To address these limitations, we present the Anchor-based Mul- timodal Embedding with Semantic Synchronization (A-MESS) framework. We first design an Anchor-based Multimodal Embed- ding (A-ME) module that employs an anchor-based embedding fusion mechanism to integrate multimodal inputs. Furthermore, we develop a Semantic Synchronization (SS) strategy with the Triplet Contrastive Learning pipeline, which optimizes the pro- cess by synchronizing multimodal representation with label de- scriptions produced by the large language model. Comprehensive experiments indicate that our A-MESS achieves state-of-the-art and provides substantial insight into multimodal representation and downstream tasks.
Abstract (translated)
在多模态意图识别（MIR）领域，目标是通过整合各种模态（如语言文本、肢体动作和语调）来识别人类的意图。然而，现有方法难以充分捕捉这些模态之间的内在联系，并且忽视了相应的语义表示。为了解决这些问题，我们提出了带有语义同步的锚点基多模态嵌入框架（Anchor-based Multimodal Embedding with Semantic Synchronization, A-MESS）。首先，我们设计了一个基于锚点的多模态嵌入模块（Anchor-based Multimodal Embedding, A-ME），该模块采用了一种基于锚点的嵌入融合机制来整合多模态输入。此外，我们还开发了语义同步策略，并结合三元组对比学习管道，通过将多模态表示与大型语言模型生成的标签描述进行同步优化，进一步改进过程。全面的实验表明，我们的A-MESS框架达到了最先进的水平，并为多模态表征和下游任务提供了重要的见解。
URL
https://arxiv.org/abs/2503.19474
