Abstract
Multimodal learning plays a pivotal role in advancing artificial intelligence systems by incorporating information from multiple modalities to build a more comprehensive representation. Despite its importance, current state-of-the-art models still suffer from severe limitations that prevent the successful development of a fully multimodal model. Such methods may not provide indicators that all the involved modalities are effectively aligned. As a result, some modalities may not be aligned, undermining the effectiveness of the model in downstream tasks where multiple modalities should provide additional information that the model fails to exploit. In this paper, we present TRIANGLE: TRI-modAl Neural Geometric LEarning, the novel proposed similarity measure that is directly computed in the higher-dimensional space spanned by the modality embeddings. TRIANGLE improves the joint alignment of three modalities via a triangle-area similarity, avoiding additional fusion layers or pairwise similarities. When incorporated in contrastive losses replacing cosine similarity, TRIANGLE significantly boosts the performance of multimodal modeling, while yielding interpretable alignment rationales. Extensive evaluation in three-modal tasks such as video-text and audio-text retrieval or audio-video classification, demonstrates that TRIANGLE achieves state-of-the-art results across different datasets improving the performance of cosine-based methods up to 9 points of Recall@1.
Abstract (translated)
多模态学习在通过融合来自多种模式的信息来构建更全面的表示，从而推进人工智能系统方面发挥着核心作用。尽管其重要性不言而喻，但目前最先进的模型仍然受到严重影响性能限制的问题，这些问题阻碍了完全多模态模型的成功开发。现有方法可能无法提供所有涉及模式有效对齐的指标。因此，在下游任务中，某些模式可能未能充分对齐，这些任务要求多种模式共同提供额外信息以增强模型性能，而当前模型往往未能充分利用这一点。 在本文中，我们提出了TRIANGLE：TRI模态神经几何学习，这是一种新颖的相似度测量方法，可以直接计算出现在由模式嵌入所定义的高维空间中的三角形面积。通过采用这种三角形面积相似性，TRIANGLE改进了三种模式的同时对齐，从而避免了额外融合层或成对相似性的使用。当在对比损失中替换余弦相似性时，TRIANGLE显著增强了多模态建模性能，并提供了可解释的对齐原理。 在视频-文本、音频-文本检索以及音频-视频分类等三模式任务中的广泛评估表明，TRIANGLE在各种数据集上实现了最先进的结果，在基于余弦的方法的基础上将Recall@1提高了高达9个百分点。
URL
https://arxiv.org/abs/2509.24734
