Abstract
Existing large video-language models (LVLMs) struggle to comprehend long videos correctly due to limited context. To address this problem, fine-tuning long-context LVLMs and employing GPT-based agents have emerged as promising solutions. However, fine-tuning LVLMs would require extensive high-quality data and substantial GPU resources, while GPT-based agents would rely on proprietary models (e.g., GPT-4o). In this paper, we propose Video Retrieval-Augmented Generation (Video-RAG), a training-free and cost-effective pipeline that employs visually-aligned auxiliary texts to help facilitate cross-modality alignment while providing additional information beyond the visual content. Specifically, we leverage open-source external tools to extract visually-aligned information from pure video data (e.g., audio, optical character, and object detection), and incorporate the extracted information into an existing LVLM as auxiliary texts, alongside video frames and queries, in a plug-and-play manner. Our Video-RAG offers several key advantages: (i) lightweight with low computing overhead due to single-turn retrieval; (ii) easy implementation and compatibility with any LVLM; and (iii) significant, consistent performance gains across long video understanding benchmarks, including Video-MME, MLVU, and LongVideoBench. Notably, our model demonstrates superior performance over proprietary models like Gemini-1.5-Pro and GPT-4o when utilized with a 72B model.
Abstract (translated)
现有的大型视频语言模型（LVLMs）由于上下文限制，难以准确理解长视频。为了解决这个问题，微调具有长上下文的LVLM和使用基于GPT的代理被认为是很有前景的解决方案。然而，微调LVLM需要大量的高质量数据和显著的GPU资源，而基于GPT的代理则依赖于专有的模型（例如，GPT-4o）。在本文中，我们提出了视频检索增强生成（Video-RAG），这是一种无需训练且成本效益高的流程，通过使用视觉对齐的辅助文本帮助跨模态对齐，并提供超出视觉内容之外的信息。具体而言，我们利用开源外部工具从纯视频数据中提取视觉对齐信息（例如，音频、光学字符和目标检测），并将提取到的信息以插件方式作为辅助文本与视频帧和查询一起集成到现有的LVLM中。我们的Video-RAG提供了几个关键优势：(i)由于单次检索而计算开销低；(ii)易于实现，并且可以兼容任何LVLM；以及(iii)在长视频理解基准测试（如Video-MME、MLVU和LongVideoBench）上表现出显著且一致的性能提升。值得注意的是，当使用72B模型时，我们的模型在性能上优于专有模型Gemini-1.5-Pro和GPT-4o。
URL
https://arxiv.org/abs/2411.13093
