Abstract
The rise of large-scale multimodal models has paved the pathway for groundbreaking advances in generative modeling and reasoning, unlocking transformative applications in a variety of complex tasks. However, a pressing question that remains is their genuine capability for stronger forms of generalization, which has been largely underexplored in the multimodal setting. Our study aims to address this by examining sequential compositional generalization using \textsc{CompAct} (\underline{Comp}ositional \underline{Act}ivities)\footnote{Project Page: \url{this http URL}}, a carefully constructed, perceptually grounded dataset set within a rich backdrop of egocentric kitchen activity videos. Each instance in our dataset is represented with a combination of raw video footage, naturally occurring sound, and crowd-sourced step-by-step descriptions. More importantly, our setup ensures that the individual concepts are consistently distributed across training and evaluation sets, while their compositions are novel in the evaluation set. We conduct a comprehensive assessment of several unimodal and multimodal models. Our findings reveal that bi-modal and tri-modal models exhibit a clear edge over their text-only counterparts. This highlights the importance of multimodality while charting a trajectory for future research in this domain.
Abstract (translated)
大规模多模态模型的兴起为生成建模和推理的突破奠定了道路，同时也为各种复杂任务中发现了 transformative 应用。然而，一个仍然需要解决的问题是对更强的泛化能力的真正能力，这在多模态环境中已经没有被深入探讨。我们的研究旨在通过研究使用 \textsc{CompAct} (\underline{Comp}ositional \underline{Act}ities)\footnote{项目页面：\url{这个链接}）来解决这一问题，这是一个精心构建的、基于感知基础的丰富背景的元场景视频数据集。我们数据集中的每个实例都用原始视频 footage、自然产生的声音和来自众包的逐步描述来表示。更重要的是，我们的设置确保了单个概念在训练和评估集中是一致的，而它们的组合在评估集上是新颖的。我们对几个单模态和多模态模型进行了全面的评估。我们的研究结果表明，双模态和三模态模型在文本仅有的模型方面具有明显的优势。这突出了在研究这个领域未来发展中多模态的重要性。
URL
https://arxiv.org/abs/2404.12013
