Abstract
The development of Large Vision-Language Models (LVLMs) is striving to catch up with the success of Large Language Models (LLMs), yet it faces more challenges to be resolved. Very recent works enable LVLMs to localize object-level visual contents and ground text to them. Nonetheless, current LVLMs still struggle to precisely understand visual relations due to the lack of relevant data. In this work, we present RelationVLM, a large vision-language model capable of comprehending various levels and types of relations whether across multiple images or within a video. Specifically, we devise a multi-stage relation-aware training scheme and a series of corresponding data configuration strategies to bestow RelationVLM with the capabilities of understanding semantic relations, temporal associations and geometric transforms. Extensive case studies and quantitative evaluations show RelationVLM has strong capability in understanding such relations and emerges impressive in-context capability of reasoning from few-shot examples by comparison. This work fosters the advancements of LVLMs by enabling them to support a wider range of downstream applications toward artificial general intelligence.
Abstract (translated)
大视图语言模型的（LVLM）发展试图赶超大型语言模型（LLMs）的成功，然而要解决的问题还很多。最近的工作使LVLM能够将物体级视觉内容进行本地化，并将文本与它们绑定。然而，由于缺乏相关数据，当前的LVLM仍然很难精确理解视觉关系。在本文中，我们提出了关系VLM，一种大型视觉语言模型，可以理解各种关系，无论是跨越多张图片还是在一个视频中。具体来说，我们设计了一个多阶段关系感知训练计划和一系列相应的数据配置策略，以赋予关系VLM理解语义关系、时间关联和几何变换的能力。大量案例研究和定量评估表明，关系VLM在理解这些关系方面具有很强的能力，并且在从少样本情况下进行推理时表现出色。这项工作通过使LVLM支持更广泛的下游应用，促进了其发展。
URL
https://arxiv.org/abs/2403.12801
