Abstract
Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks (RNNs) and hybrid models that mix recurrence and attention. Yet there is no consensus on whether the potential benefits of these new architectures justify the risk and effort of scaling them up. To address this, we provide evidence for the advantages of hybrid models over pure transformers on several fronts. First, theoretically, we show that hybrid models do not merely inherit the expressivity of transformers and linear RNNs, but can express tasks beyond both, such as code execution. Putting this theory to practice, we train Olmo Hybrid, a 7B-parameter model largely comparable to Olmo 3 7B but with the sliding window layers replaced by Gated DeltaNet layers. We show that Olmo Hybrid outperforms Olmo 3 across standard pretraining and mid-training evaluations, demonstrating the benefit of hybrid models in a controlled, large-scale setting. We find that the hybrid model scales significantly more efficiently than the transformer, explaining its higher performance. However, its unclear why greater expressivity on specific formal problems should result in better scaling or superior performance on downstream tasks unrelated to those problems. To explain this apparent gap, we return to theory and argue why increased expressivity should translate to better scaling efficiency, completing the loop. Overall, our results suggest that hybrid models mixing attention and recurrent layers are a powerful extension to the language modeling paradigm: not merely to reduce memory during inference, but as a fundamental way to obtain more expressive models that scale better during pretraining.
Abstract (translated)
近期研究揭示了非Transformer语言模型的潜力，尤其是线性循环神经网络（RNNs）以及混合循环与注意力机制的混合模型。然而，学界尚未就这些新架构的潜在优势是否值得承担其扩展的风险与成本达成共识。为探讨此问题，我们从多个方面提供了混合模型优于纯Transformer的证据。首先，理论上我们证明混合模型不仅继承了Transformer与线性RNN的表达能力，更能表达两者无法处理的任务（如代码执行）。将理论付诸实践，我们训练了Olmo Hybrid——一个70亿参数模型，其架构与Olmo 3 7B高度相似，但将滑动窗口层替换为门控DeltaNet层。实验表明，Olmo Hybrid在标准预训练与中期评估中均超越Olmo 3，证实了混合模型在受控大规模场景中的优势。我们发现混合模型的扩展效率显著高于Transformer，这解释了其更优性能。然而，为何在特定形式化问题上更强的表达能力能转化为与这些问题无关的下游任务中更好的扩展性或性能，仍不明确。为解释这一明显矛盾，我们回归理论论证增强的表达能力如何转化为更优的扩展效率，从而形成完整逻辑闭环。总体而言，我们的结果表明，混合注意力与循环层的混合模型是对语言建模范式的有力拓展：其价值不仅在于降低推理时的内存消耗，更在于提供了一种获取更强表达能力、实现更优预训练扩展性的根本性途径。
URL
https://arxiv.org/abs/2604.03444
