Abstract
Video question answering (VideoQA) is a challenging task that requires integrating spatial, temporal, and semantic information to capture the complex dynamics of video sequences. Although recent advances have introduced various approaches for video understanding, most existing methods still rely on locating relevant frames to answer questions rather than reasoning through the evolving storyline as humans do. Humans naturally interpret videos through coherent storylines, an ability that is crucial for making robust and contextually grounded predictions. To address this gap, we propose SVAgent, a storyline-guided cross-modal multi-agent framework for VideoQA. The storyline agent progressively constructs a narrative representation based on frames suggested by a refinement suggestion agent that analyzes historical failures. In addition, cross-modal decision agents independently predict answers from visual and textual modalities under the guidance of the evolving storyline. Their outputs are then evaluated by a meta-agent to align cross-modal predictions and enhance reasoning robustness and answer consistency. Experimental results demonstrate that SVAgent achieves superior performance and interpretability by emulating human-like storyline reasoning in video understanding.
Abstract (translated)
视频问答（VideoQA）是一项具有挑战性的任务，需要整合空间、时序和语义信息以捕捉视频序列的复杂动态。尽管近期研究已提出多种视频理解方法，但现有技术仍主要依赖定位相关帧来回答问题，而非像人类一样通过演变的故事情节进行推理。人类天然通过连贯的叙事来解读视频，这种能力对于做出稳健且基于上下文的预测至关重要。为弥补这一差距，我们提出了SVAgent——一个叙事引导的跨模态多智能体视频问答框架。其中，叙事智能体基于由分析历史失败案例的优化建议智能体所推荐的帧，逐步构建叙事表征；同时，跨模态决策智能体在演变的叙事引导下，分别从视觉和文本模态独立预测答案。随后，元智能体评估这些输出以对齐跨模态预测，并增强推理稳健性与答案一致性。实验结果表明，SVAgent通过模拟人类式的叙事推理，在视频理解中实现了卓越性能和可解释性。
URL
https://arxiv.org/abs/2604.05079
