Abstract
Video Large Language Models (Video-LLMs) require continual learning to adapt to non-stationary real-world data. However, existing benchmarks fall short of evaluating modern foundation models: many still rely on models without large-scale pre-training, and prevailing benchmarks typically partition a single dataset into sub-tasks, resulting in high task redundancy and negligible forgetting on pre-trained Video-LLMs. To address these limitations, we propose CL-VISTA, a benchmark tailored for continual video understanding of Video-LLMs. By curating 8 diverse tasks spanning perception, understanding, and reasoning, CL-VISTA induces substantial distribution shifts that effectively expose catastrophic forgetting. To systematically assess CL methods, we establish a comprehensive evaluation framework comprising 6 distinct protocols across 3 critical dimensions: performance, computational efficiency, and memory footprint. Notably, the performance dimension incorporates a general video understanding assessment to assess whether CL methods genuinely enhance foundational intelligence or merely induce task-specific overfitting. Extensive benchmarking of 10 mainstream CL methods reveals a fundamental trade-off: no single approach achieves universal superiority across all dimensions. Methods that successfully mitigate catastrophic forgetting tend to compromise generalization or incur prohibitive computational and memory overheads. We hope CL-VISTA provides critical insights for advancing continual learning in multimodal foundation models.
Abstract (translated)
视频大语言模型需要持续学习以适应非平稳的真实世界数据。然而，现有基准测试在评估现代基础模型时存在不足：许多仍依赖未经大规模预训练的模型，且主流基准通常将单一数据集划分为多个子任务，导致任务冗余度高，对预训练的视频大语言模型几乎不产生遗忘效应。为应对这些局限，我们提出了CL-VISTA——一个专为视频大语言模型持续视频理解设计的基准测试。通过精心选取涵盖感知、理解与推理三大维度的8项多样化任务，CL-VISTA能引发显著的数据分布偏移，从而有效暴露灾难性遗忘现象。为系统评估持续学习方法，我们建立了包含性能、计算效率与内存占用三个关键维度的6种评估协议的综合框架。其中性能维度特别引入通用视频理解评估，以判断持续学习方法是否真正提升基础智能，还是仅导致任务特定过拟合。对10种主流持续学习方法的大规模基准测试揭示了一个根本性权衡：没有任何单一方法能在所有维度实现全面优势。成功缓解灾难性遗忘的方法往往以牺牲泛化能力或产生高昂的计算与内存开销为代价。我们希望CL-VISTA能为推动多模态基础模型的持续学习研究提供关键洞见。
URL
https://arxiv.org/abs/2604.00677
