Abstract
Large Vision-Language Models often generate hallucinated content that is not grounded in its visual inputs. While prior work focuses on mitigating hallucinations, we instead explore leveraging hallucination correction as a training objective to improve video-language alignment. We introduce HACA, a self-training framework learning to correct hallucinations in descriptions that do not align with the video content. By identifying and correcting inconsistencies, HACA enhances the model's ability to align video and textual representations for spatio-temporal reasoning. Our experimental results show consistent gains in video-caption binding and text-to-video retrieval tasks, demonstrating that hallucination correction-inspired tasks serve as an effective strategy for improving vision and language alignment.
Abstract (translated)
大型视觉-语言模型常常会产生与视觉输入不一致的虚构内容。虽然之前的研究侧重于减少这种虚构现象，但我们转而探索将虚构修正作为训练目标来提高视频和文本之间的对齐能力。我们提出了HACA（Hallucination Correction for Video-Language Alignment），这是一种自我训练框架，用于纠正那些不符合视频内容描述中的虚构部分。通过识别并纠正不一致之处，HACA增强了模型在时空推理中将视频与文本表示进行对齐的能力。实验结果表明，在视频字幕绑定和文字到视频检索任务上，我们的方法取得了持续的性能提升，证明了以虚构修正为灵感的任务是一种有效提高视觉和语言对齐能力的策略。
URL
https://arxiv.org/abs/2502.15079
