Abstract
Recent vision and multimodal foundation backbones, such as Transformer families and state-space models like Mamba, have achieved remarkable progress, enabling unified modeling across images, text, and beyond. Despite their empirical success, these architectures remain far from the computational principles of the human brain, often demanding enormous amounts of training data while offering limited interpretability. In this work, we propose the Vision Hopfield Memory Network (V-HMN), a brain-inspired foundation backbone that integrates hierarchical memory mechanisms with iterative refinement updates. Specifically, V-HMN incorporates local Hopfield modules that provide associative memory dynamics at the image patch level, global Hopfield modules that function as episodic memory for contextual modulation, and a predictive-coding-inspired refinement rule for iterative error correction. By organizing these memory-based modules hierarchically, V-HMN captures both local and global dynamics in a unified framework. Memory retrieval exposes the relationship between inputs and stored patterns, making decisions more interpretable, while the reuse of stored patterns improves data efficiency. This brain-inspired design therefore enhances interpretability and data efficiency beyond existing self-attention- or state-space-based approaches. We conducted extensive experiments on public computer vision benchmarks, and V-HMN achieved competitive results against widely adopted backbone architectures, while offering better interpretability, higher data efficiency, and stronger biological plausibility. These findings highlight the potential of V-HMN to serve as a next-generation vision foundation model, while also providing a generalizable blueprint for multimodal backbones in domains such as text and audio, thereby bridging brain-inspired computation with large-scale machine learning.
Abstract (translated)
近期视觉与多模态基础主干网络，如Transformer系列及Mamba等状态空间模型，已取得显著进展，实现了图像、文本等多模态的统一建模。尽管这些架构在经验上取得成功，但其设计原理仍与人类大脑的计算机制相去甚远，往往需要海量训练数据，且可解释性有限。本研究提出脑启发式基础主干网络——视觉霍普菲尔德记忆网络（V-HMN），该网络将分层记忆机制与迭代细化更新相结合。具体而言，V-HMN集成了局部霍普菲尔德模块（在图像块层面提供联想记忆动态）、全局霍普菲尔德模块（作为情景记忆实现上下文调制），以及受预测编码启发的迭代纠错细化规则。通过将这些基于记忆的模块进行分层组织，V-HMN在统一框架内同时捕捉局部与全局动态。记忆检索过程揭示了输入与存储模式之间的关系，使决策更具可解释性；而存储模式的重用则提升了数据效率。因此，这种脑启发式设计在可解释性与数据效率方面超越了现有的自注意力或状态空间方法。我们在公开计算机视觉基准上进行了大量实验，结果表明V-HMN在性能上可与广泛采用的主干架构相竞争，同时提供了更好的可解释性、更高的数据效率以及更强的生物学合理性。这些发现凸显了V-HMN作为下一代视觉基础模型的潜力，并为文本、音频等领域的多模态主干网络提供了可推广的通用框架，从而在脑启发式计算与大规模机器学习之间架起桥梁。
URL
https://arxiv.org/abs/2603.25157
