Abstract
Long video question answering is a challenging task that involves recognizing short-term activities and reasoning about their fine-grained relationships. State-of-the-art video Large Language Models (vLLMs) hold promise as a viable solution due to their demonstrated emergent capabilities on new tasks. However, despite being trained on millions of short seconds-long videos, vLLMs are unable to understand minutes-long videos and accurately answer questions about them. To address this limitation, we propose a lightweight and self-supervised approach, Key frame-conditioned long video-LLM (Koala), that introduces learnable spatiotemporal queries to adapt pretrained vLLMs for generalizing to longer videos. Our approach introduces two new tokenizers that condition on visual tokens computed from sparse video key frames for understanding short and long video moments. We train our proposed approach on HowTo100M and demonstrate its effectiveness on zero-shot long video understanding benchmarks, where it outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across all tasks. Surprisingly, we also empirically show that our approach not only helps a pretrained vLLM to understand long videos but also improves its accuracy on short-term action recognition.
Abstract (translated)
长的视频问答是一个具有挑战性的任务，需要识别短期的活动并进行关于它们细粒度关系的推理。先进的视频大型语言模型（vLLMs）因其在新的任务上表现出的新兴能力而具有实现的可能。然而，尽管它们通过训练掌握了数百万秒长的视频，但vLLMs无法理解几分钟长的视频，也无法准确回答关于它们的问题。为了解决这个问题，我们提出了一个轻量级且自监督的方法：关键帧条件下的长视频-LLM（Koala），它引入了可学习的时态和空间查询，以将预训练的vLLM扩展到更长的视频中。我们的方法引入了两个新的词标器，它们基于从稀疏视频关键帧计算的视觉标记物来理解短和长视频时刻。我们在HowTo100M上训练我们的方法，并在零散的视频理解基准上证明了其有效性，其中它在所有任务上的绝对准确度比最先进的模型高3 - 6%。令人惊讶的是，我们还通过实验发现，我们的方法不仅有助于预训练的vLLM理解长视频，而且还有助于提高其对短期动作识别的准确性。
URL
https://arxiv.org/abs/2404.04346
