Abstract
Integrating vision models into large language models (LLMs) has sparked significant interest in creating vision-language foundation models, especially for video understanding. Recent methods often utilize memory banks to handle untrimmed videos for video-level understanding. However, they typically compress visual memory using similarity-based greedy approaches, which can overlook the contextual importance of individual tokens. To address this, we introduce an efficient LLM adapter designed for video-level understanding of untrimmed videos that prioritizes the contextual relevance of spatio-temporal tokens. Our framework leverages scorer networks to selectively compress the visual memory bank and filter spatial tokens based on relevance, using a differentiable Top-K operator for end-to-end training. Across three key video-level understanding tasks$\unicode{x2013}$ untrimmed video classification, video question answering, and video captioning$\unicode{x2013}$our method achieves competitive or superior results on four large-scale datasets while reducing computational overhead by up to 34%. The code will be available soon on GitHub.
Abstract (translated)
将视觉模型整合到大型语言模型（LLMs）中，激发了创建视觉-语言基础模型的兴趣，特别是在视频理解方面。近期的方法通常利用内存银行处理未修剪的视频以进行视频级别的理解。然而，它们一般采用基于相似性的贪婪方法来压缩视觉记忆，这可能会忽视单个标记上下文的重要性。为解决这一问题，我们引入了一种高效的LLM适配器，专门用于未修剪视频的视频级别理解，并优先考虑空间-时间令牌的上下文相关性。我们的框架利用评分网络选择性地压缩视觉内存银行并根据相关性过滤空间令牌，使用可微分Top-K操作符进行端到端训练。在三个关键的视频级理解任务——未修剪视频分类、视频问答和视频字幕上，我们的方法在四个大规模数据集上取得了竞争性的或优越的结果，并且减少了最多34%的计算开销。代码不久将在GitHub上发布。
URL
https://arxiv.org/abs/2504.05491
