Abstract
Despite the growing video understanding capabilities of recent Multimodal Large Language Models (MLLMs), existing video benchmarks primarily assess understanding based on models' static, internal knowledge, rather than their ability to learn and adapt from dynamic, novel contexts from few examples. To bridge this gap, we present Demo-driven Video In-Context Learning, a novel task focused on learning from in-context demonstrations to answer questions about the target videos. Alongside this, we propose Demo-ICL-Bench, a challenging benchmark designed to evaluate demo-driven video in-context learning capabilities. Demo-ICL-Bench is constructed from 1200 instructional YouTube videos with associated questions, from which two types of demonstrations are derived: (i) summarizing video subtitles for text demonstration; and (ii) corresponding instructional videos as video demonstrations. To effectively tackle this new challenge, we develop Demo-ICL, an MLLM with a two-stage training strategy: video-supervised fine-tuning and information-assisted direct preference optimization, jointly enhancing the model's ability to learn from in-context examples. Extensive experiments with state-of-the-art MLLMs confirm the difficulty of Demo-ICL-Bench, demonstrate the effectiveness of Demo-ICL, and thereby unveil future research directions.
Abstract (translated)
尽管最近的多模态大型语言模型（MLLMs）在视频理解能力方面有所提升，但现有的视频基准测试主要基于评估模型的静态、内部知识，而不是它们从少量动态新上下文中学习和适应的能力。为了填补这一空白，我们提出了由演示驱动的视频情境学习任务，该任务专注于通过在情境中的演示来学习并回答关于目标视频的问题。与此相伴，我们还提出了一项新的挑战性基准——Demo-ICL-Bench，旨在评估由演示驱动的视频情境学习能力。 Demo-ICL-Bench 由1200个带有相关问题的教育类YouTube视频构建而成，从中派生出两种类型的示范：(i) 摘要视频字幕以生成文本示例；和(ii) 对应的教程视频作为视频示例。为了有效应对这一新挑战，我们开发了Demo-ICL，这是一种具有两阶段训练策略的MLLM：视频监督微调和信息辅助直接偏好优化，共同增强了模型从情境中例子学习的能力。 与最先进的多模态大型语言模型进行广泛实验后证实了Demo-ICL-Bench 的难度，并展示了 Demo-ICL 的有效性，从而揭示了未来的研究方向。
URL
https://arxiv.org/abs/2602.08439
