Abstract
While multimodal large language models have demonstrated impressive short-term reasoning, they struggle with long-horizon video understanding due to limited context windows and static memory mechanisms that fail to mirror human cognitive efficiency. Existing paradigms typically fall into two extremes: vision-centric methods that incur high latency and redundancy through dense visual accumulation, or text-centric approaches that suffer from detail loss and hallucination via aggressive captioning. To bridge this gap, we propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory. MM-Mem structures memory hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic Schema, enabling the progressive distillation of fine-grained perceptual traces (verbatim) into high-level semantic schemas (gist). Furthermore, to govern the dynamic construction of memory, we derive a Semantic Information Bottleneck objective and introduce SIB-GRPO to optimize the trade-off between memory compression and task-relevant information retention. In inference, we design an entropy-driven top-down memory retrieval strategy, which first tries with the abstract Symbolic Schema and progressively "drills down" to the Sensory Buffer and Episodic Stream under high uncertainty. Extensive experiments across 4 benchmarks confirm the effectiveness of MM-Mem on both offline and streaming tasks, demonstrating robust generalization and validating the effectiveness of cognition-inspired memory organization. Code is available at this https URL.
Abstract (translated)
尽管多模态大型语言模型在短期推理方面表现出色，但由于有限的上下文窗口和静态记忆机制，它们在理解长时段视频时遇到困难，这些机制无法模仿人类的认知效率。现有方法通常走向两个极端：一种是以视觉为中心的方法，通过密集的视觉积累导致高延迟和冗余；另一种是基于文本的方法，由于过分强调字幕而造成细节丢失和幻觉。 为了弥合这一差距，我们提出了MM-Mem（多模态金字塔记忆架构），该架构建立在模糊痕迹理论的基础上。MM-Mem将内存结构化为三个层级：感觉缓冲区、事件流以及符号模式，从而能够逐步提炼出细微的感知痕迹（verbatim）并将其转化为高层次的语义模型（gist）。为了动态构建内存，我们推导出了一个语义信息瓶颈目标，并引入了SIB-GRPO来优化内存压缩和任务相关信息保留之间的权衡。在推理过程中，我们设计了一种基于熵驱动的自顶向下记忆检索策略，在高度不确定性情况下，首先尝试从抽象符号模式开始，然后逐步“深入”到感觉缓冲区和事件流。 通过四个基准测试的广泛实验表明了MM-Mem在离线任务和流式任务上的有效性，展示了强大的泛化能力和认知启发的记忆组织的有效性。相关代码可在该网址上获取（请将"this https URL"替换为实际链接）。
URL
https://arxiv.org/abs/2603.01455
