Abstract
Large multimodal models (LMMs) have shown remarkable progress in audio-visual understanding, yet they struggle with real-world scenarios that require complex reasoning across extensive video collections. Existing benchmarks for video question answering remain limited in scope, typically involving one clip per query, which falls short of representing the challenges of large-scale, audio-visual retrieval and reasoning encountered in practical applications. To bridge this gap, we introduce a novel task named AV-HaystacksQA, where the goal is to identify salient segments across different videos in response to a query and link them together to generate the most informative answer. To this end, we present AVHaystacks, an audio-visual benchmark comprising 3100 annotated QA pairs designed to assess the capabilities of LMMs in multi-video retrieval and temporal grounding task. Additionally, we propose a model-agnostic, multi-agent framework MAGNET to address this challenge, achieving up to 89% and 65% relative improvements over baseline methods on BLEU@4 and GPT evaluation scores in QA task on our proposed AVHaystacks. To enable robust evaluation of multi-video retrieval and temporal grounding for optimal response generation, we introduce two new metrics, STEM, which captures alignment errors between a ground truth and a predicted step sequence and MTGS, to facilitate balanced and interpretable evaluation of segment-level grounding performance. Project: this https URL
Abstract (translated)
大型多模态模型（LMMs）在音频-视觉理解方面取得了显著的进步，但它们在处理需要跨大量视频集合进行复杂推理的真实场景时仍面临挑战。现有的视频问答基准测试范围有限，通常每个查询仅涉及一个片段，这无法代表大规模、音视检索和推理的实际应用中的挑战。为了弥合这一差距，我们引入了一个名为AV-HaystacksQA的新任务，在该任务中，目标是在响应查询时识别不同视频之间的关键段落，并将它们链接起来以生成最具有信息量的答案。为此，我们提出了一个音频-视觉基准测试AVHaystacks，包含3100个带有标注的问答对，旨在评估LMMs在多视频检索和时间定位任务中的能力。此外，为解决这一挑战，我们提出了一种模型无关、多代理框架MAGNET，在我们提出的AVHaystacks基准上的QA任务中，与基线方法相比，其BLEU@4评分提升了高达89%，GPT评价分数提升了65%。为了实现对多视频检索和时间定位的有效评估以生成最佳响应，我们引入了两个新的指标：STEM用于捕捉实际步骤序列与预测步骤序列之间的对齐错误；MTGS则有助于平衡和可解释地评估段落级定位性能。 项目链接：[此链接](https://this-url.com)
URL
https://arxiv.org/abs/2506.07016
