Abstract
Recent advances in artificial neural networks for machine learning, and language modeling in particular, have established a family of recurrent neural network (RNN) architectures that, unlike conventional RNNs with vector-form hidden states, use two-dimensional (2D) matrix-form hidden states. Such 2D-state RNNs, known as Fast Weight Programmers (FWPs), can be interpreted as a neural network whose synaptic weights (called fast weights) dynamically change over time as a function of input observations, and serve as short-term memory storage; corresponding synaptic weight modifications are controlled or programmed by another network (the programmer) whose parameters are trained (e.g., by gradient descent). In this Primer, we review the technical foundations of FWPs, their computational characteristics, and their connections to transformers and state space models. We also discuss connections between FWPs and models of synaptic plasticity in the brain, suggesting a convergence of natural and artificial intelligence.
Abstract (translated)
最近在机器学习领域，特别是在语言模型方面的进展，确立了一类循环神经网络（RNN）架构。与传统的使用向量形式隐状态的RNN不同，这类新架构采用二维矩阵形式的隐藏状态。这种具有2D状态的RNN被称为快速权重编程器（FWPs），可以被解释为一个其突触权重（称为快权重）会随输入观察结果动态变化的神经网络，并且这些权重作为短期记忆存储；相应的突触权重调整由另一个网络（即编程器）控制，该网络的参数通过训练获得（例如，通过梯度下降法）。在本文中，我们回顾了FWPs的技术基础、计算特性以及它们与变压器和状态空间模型之间的联系。此外，我们还讨论了FWPs与大脑突触可塑性模型之间的关联，这表明自然智能和人工智能正在趋于融合。
URL
https://arxiv.org/abs/2508.08435
