Abstract
Video temporal grounding (VTG) is a critical task in video understanding and a key capability for extending video large language models (Vid-LLMs) to broader applications. However, existing Vid-LLMs rely on uniform frame sampling to extract video information, resulting in a sparse distribution of key frames and the loss of crucial temporal cues. To address this limitation, we propose Grounded Visual Token Sampling (GroundVTS), a Vid-LLM architecture that focuses on the most informative temporal segments. GroundVTS employs a fine-grained, query-guided mechanism to filter visual tokens before feeding them into the LLM, thereby preserving essential spatio-temporal information and maintaining temporal coherence. Futhermore, we introduce a progressive optimization strategy that enables the LLM to effectively adapt to the non-uniform distribution of visual features, enhancing its ability to model temporal dependencies and achieve precise video localization. We comprehensively evaluate GroundVTS on three standard VTG benchmarks, where it outperforms existing methods, achieving a 7.7-point improvement in mIoU for moment retrieval and 12.0-point improvement in mAP for highlight detection. Code is available at this https URL.
Abstract (translated)
视频时间定位（VTG）是视频理解中的关键任务，也是扩展视频大语言模型（Vid-LLMs）应用范围的核心能力。然而，现有Vid-LLMs依赖均匀帧采样提取视频信息，导致关键帧分布稀疏并丢失重要时间线索。针对这一局限，我们提出了Grounded Visual Token Sampling（GroundVTS），一种聚焦于最具信息量时间片段的Vid-LLM架构。GroundVTS采用细粒度、查询引导的机制，在将视觉标记输入大语言模型前进行筛选，从而保留关键时空信息并维持时间连贯性。此外，我们引入了渐进式优化策略，使大语言模型能够有效适应视觉特征的非均匀分布，增强其建模时间依赖关系的能力，实现精准的视频定位。我们在三个标准VTG基准测试上对GroundVTS进行了全面评估，其性能超越现有方法，在时刻检索任务中mIoU提升7.7点，在亮点检测任务中mAP提升12.0点。代码已公开于此https链接。
URL
https://arxiv.org/abs/2604.02093
