Abstract
The ability to understand long videos is vital for embodied intelligent agents, because their effectiveness depends on how well they can accumulate, organize, and leverage long-horizon perceptual memories. Recently, multimodal LLMs have been gaining popularity for solving the long video understanding task due to their general ability to understand natural language and to leverage world knowledge. However, as the number of frames provided to an MLLM increases, the quality of its responses tends to degrade, and inference time grows. Therefore, when using MLLMs for long video understanding, a crucial step is selecting key frames from the video to answer user queries. In this work, we develop FocusGraph, a framework for keyframe selection for question answering over long egocentric videos. It leverages a lightweight trainable Scene-Caption LLM Selector that selects query-relevant clips based on their graph-based captions, and a training-free method for selecting keyframes from these clips. Unlike existing methods, the proposed Scene-Caption LLM Selector does not rely on the original sequence of low-resolution frames; instead, it operates on a compact textual representation of the scene. We then design a training-free Patch-wise Sparse-Flow Retention (PSFR) method to select keyframes from the resulting sequence of clips, which are fed into an MLLM to produce the final answer. Together, these components enable FocusGraph to achieve state-of-the-art results on challenging egocentric long-video question answering benchmarks, including FindingDory and HourVideo, while significantly reducing inference time relative to baseline approaches.
Abstract (translated)
理解长视频对于具身智能代理来说至关重要，因为它们的效率取决于积累、组织和利用长期感知记忆的能力。最近，多模态大语言模型（MLLM）由于其理解和利用世界知识以及自然语言的一般能力，在解决长视频理解任务方面越来越受欢迎。然而，随着提供给MLLM的帧数增加，其响应质量往往会下降，并且推理时间也会增长。因此，在使用MLLM进行长视频理解时，关键步骤之一是从视频中选择与用户查询相关的关键帧。 在这项工作中，我们开发了FocusGraph，这是一个用于在长时间第一人称视角视频问答中选择关键帧的框架。它利用了一个轻量级可训练的场景描述大语言模型选择器（Scene-Caption LLM Selector），该选择器基于基于图的字幕选择与查询相关的片段，并且还设计了一种无需训练的方法从这些片段中选择关键帧。与现有方法不同，所提出的场景描述大语言模型选择器不依赖于原始低分辨率帧序列；相反，它在紧凑的文本形式的场景表示上运行。然后，我们设计了一种无训练的分块稀疏流保留（Patch-wise Sparse-Flow Retention, PSFR）方法来从生成的片段序列中选择关键帧，并将这些关键帧输入到一个MLLM中以产生最终答案。 通过结合这些组件，FocusGraph能够在具有挑战性的第一人称视角长视频问答基准测试（如FindingDory和HourVideo）上实现最先进的结果，同时相对于基线方法显著减少推理时间。
URL
https://arxiv.org/abs/2603.04349
