Abstract
Standard attention stores keys/values losslessly but reads them via a per-head convex average, blocking channel-wise selection. We propose the Free Energy Mixer (FEM): a free-energy (log-sum-exp) read that applies a value-driven, per-channel log-linear tilt to a fast prior (e.g., from queries/keys in standard attention) over indices. Unlike methods that attempt to improve and enrich the $(q,k)$ scoring distribution, FEM treats it as a prior and yields a value-aware posterior read at unchanged complexity, smoothly moving from averaging to per-channel selection as the learnable inverse temperature increases, while still preserving parallelism and the original asymptotic complexity ($O(T^2)$ for softmax; $O(T)$ for linearizable variants). We instantiate a two-level gated FEM that is plug-and-play with standard and linear attention, linear RNNs and SSMs. It consistently outperforms strong baselines on NLP, vision, and time-series at matched parameter budgets.
Abstract (translated)
标准注意力机制无损地存储键值，但通过每个头的凸平均读取它们，这阻碍了通道级别的选择。我们提出了自由能量混合器（FEM）：一种基于自由能（对数和指数函数）的读取方法，它为快速先验（例如来自标准注意中的查询/键的优先级）提供了一个由值驱动、每通道的对数线性倾斜。与试图改进和丰富$(q,k)$评分分布的方法不同，FEM将其视为一种先验，并在不改变复杂度的情况下生成一个值感知的后读取，随着可学习逆温度的增加，它从平均操作平滑过渡到每通道选择，同时仍然保持并行性和原有的渐近复杂度（对于softmax为$O(T^2)$；线性变体为$O(T)$）。 我们实现了一个两层门控FEM，它可以与标准注意、线性注意、线性RNN和SSM无缝集成。在匹配的参数预算下，它在自然语言处理、视觉以及时间序列任务上始终优于强大的基线方法。 这段翻译解释了Free Energy Mixer（自由能量混合器）的概念及其相对于传统注意力机制的优势，并描述了它的实现方式及应用场景中的性能表现。
URL
https://arxiv.org/abs/2602.07160
