Abstract
While datasets for video understanding have scaled to hour-long durations, they typically consist of densely concatenated clips that differ from natural, unscripted daily life. To bridge this gap, we introduce MM-Lifelong, a dataset designed for Multimodal Lifelong Understanding. Comprising 181.1 hours of footage, it is structured across Day, Week, and Month scales to capture varying temporal densities. Extensive evaluations reveal two critical failure modes in current paradigms: end-to-end MLLMs suffer from a Working Memory Bottleneck due to context saturation, while representative agentic baselines experience Global Localization Collapse when navigating sparse, month-long timelines. To address this, we propose the Recursive Multimodal Agent (ReMA), which employs dynamic memory management to iteratively update a recursive belief state, significantly outperforming existing methods. Finally, we establish dataset splits designed to isolate temporal and domain biases, providing a rigorous foundation for future research in supervised learning and out-of-distribution generalization.
Abstract (translated)
尽管视频理解的数据集已经扩展到长达数小时的时长，但这些数据集通常由密集连接的片段组成，这与自然、无剧本的日常生活存在差异。为了弥合这一差距，我们引入了MM-Lifelong数据集，该数据集旨在用于多模态终身理解。它包含181.1小时的视频资料，并按天、周和月的时间尺度结构化，以捕捉不同时间密度的变化。全面评估揭示了当前范式中的两个关键失败模式：端到端MLLM（多模态终身学习模型）因上下文饱和而遭受工作记忆瓶颈问题；代表性代理基线在处理稀疏的长达一个月的时间线上则遭遇全局定位崩溃问题。 为了解决这些问题，我们提出了一种递归多模态智能体(Recursive Multimodal Agent, ReMA)，该智能体采用动态内存管理技术来迭代更新一个递归信念状态，从而显著优于现有的方法。最后，我们制定了数据集分割方案以隔离时间和领域偏见，为监督学习和分布外泛化研究提供了一个严格的理论基础。
URL
https://arxiv.org/abs/2603.05484
