Abstract
Multimodal representation learning, with contrastive learning, plays an important role in the artificial intelligence domain. As an important subfield, video-language representation learning focuses on learning representations using global semantic interactions between pre-defined video-text pairs. However, to enhance and refine such coarse-grained global interactions, more detailed interactions are necessary for fine-grained multimodal learning. In this study, we introduce a new approach that models video-text as game players using multivariate cooperative game theory to handle uncertainty during fine-grained semantic interactions with diverse granularity, flexible combination, and vague intensity. Specifically, we design the Hierarchical Banzhaf Interaction to simulate the fine-grained correspondence between video clips and textual words from hierarchical perspectives. Furthermore, to mitigate the bias in calculations within Banzhaf Interaction, we propose reconstructing the representation through a fusion of single-modal and cross-modal components. This reconstructed representation ensures fine granularity comparable to that of the single-modal representation, while also preserving the adaptive encoding characteristics of cross-modal representation. Additionally, we extend our original structure into a flexible encoder-decoder framework, enabling the model to adapt to various downstream tasks. Extensive experiments on commonly used text-video retrieval, video-question answering, and video captioning benchmarks, with superior performance, validate the effectiveness and generalization of our method.
Abstract (translated)
多模态表示学习，特别是在对比学习中的应用，在人工智能领域扮演着重要角色。作为其重要的子领域之一，视频-语言表示学习侧重于利用预定义的视频-文本对之间的全局语义交互来学习表示形式。然而，为了增强和细化这种粗粒度的整体互动，需要更详细的互动以进行细粒度多模态学习。在这项研究中，我们引入了一种新方法，该方法采用多元合作博弈理论将视频-文本视为游戏参与者，以此处理在具有多样化粒度、灵活组合以及模糊强度的细粒度语义交互过程中的不确定性。 具体而言，我们设计了层次化的Banzhaf互作模型（Hierarchical Banzhaf Interaction），从多层级视角模拟视频片段与文本词汇之间的细粒度对应关系。此外，为了解决Banzhaf互作计算中的偏置问题，我们提出了通过单模态和跨模态组件融合来重构表示形式的方法。这种重构的表示既保持了与单模态表示相当的细粒度级别，又保留了跨模态表示适应性编码的特点。 此外，我们将原始结构扩展到一个灵活的编码-解码框架中，使模型能够适应各种下游任务。在常用的文本-视频检索、视频问答和视频描述基准测试中的广泛实验表明，我们的方法具有优越的表现力与泛化能力。
URL
https://arxiv.org/abs/2412.20964
