Abstract
Large language models (LLMs) have demonstrated exceptional capabilities in text understanding, which has paved the way for their expansion into video LLMs (Vid-LLMs) to analyze video data. However, current Vid-LLMs struggle to simultaneously retain high-quality frame-level semantic information (i.e., a sufficient number of tokens per frame) and comprehensive video-level temporal information (i.e., an adequate number of sampled frames per video). This limitation hinders the advancement of Vid-LLMs towards fine-grained video understanding. To address this issue, we introduce the SlowFocus mechanism, which significantly enhances the equivalent sampling frequency without compromising the quality of frame-level visual tokens. SlowFocus begins by identifying the query-related temporal segment based on the posed question, then performs dense sampling on this segment to extract local high-frequency features. A multi-frequency mixing attention module is further leveraged to aggregate these local high-frequency details with global low-frequency contexts for enhanced temporal comprehension. Additionally, to tailor Vid-LLMs to this innovative mechanism, we introduce a set of training strategies aimed at bolstering both temporal grounding and detailed temporal reasoning capabilities. Furthermore, we establish FineAction-CGR, a benchmark specifically devised to assess the ability of Vid-LLMs to process fine-grained temporal understanding tasks. Comprehensive experiments demonstrate the superiority of our mechanism across both existing public video understanding benchmarks and our proposed FineAction-CGR.
Abstract (translated)
大型语言模型（LLM）已经在文本理解方面表现出色，这为它们扩展到视频LLM（Vid-LLM），以分析视频数据铺平了道路。然而，当前的Vid-LLM在同时保留高质量的帧级语义信息（即每帧有足够的token数量）和全面的视频级时间信息（即每个视频采样的足够多的帧数）方面遇到困难。这种限制阻碍了Vid-LLMs向细粒度视频理解的进步。为了解决这个问题，我们引入了SlowFocus机制，该机制显著提高了等效采样频率，同时不损害帧级视觉token的质量。SlowFocus首先根据提出的问题确定查询相关的时间段，然后在此时间段上进行密集采样以提取局部高频特征。进一步利用多频混合注意模块将这些局部高频细节与全局低频上下文聚合起来，从而增强时间理解能力。此外，为了使Vid-LLM适应这种创新机制，我们还引入了一系列旨在加强时间和详细时间推理能力的训练策略。另外，我们建立了一个基准测试FineAction-CGR，专门用于评估Vid-LLMs处理细粒度时间理解任务的能力。全面的实验表明，在现有的公开视频理解基准和我们提出的FineAction-CGR上，我们的机制都表现出了优越性。
URL
https://arxiv.org/abs/2602.03589
