Abstract
Pre-Layer Normalization (Pre-LN) is the de facto choice for large language models (LLMs) and is crucial for stable pretraining and effective transfer learning. However, Pre-LN is inefficient due to repeated statistical calculations and suffers from the curse of depth. As layers grow, the magnitude and variance of the hidden state escalate, destabilizing training. Efficiency-oriented normalization-free methods such as Dynamic Tanh (DyT) improve speed but remain fragile at depth. To jointly address stability and efficiency, we propose Bounded Hyperbolic Tanh (BHyT), a drop-in replacement for Pre-LN. BHyT couples a tanh nonlinearity with explicit, data-driven input bounding to keep activations within a non-saturating range. It prevents depth-wise growth in activation magnitude and variance and comes with a theoretical stability guarantee. For efficiency, BHyT computes exact statistics once per block and replaces a second normalization with a lightweight variance approximation, enhancing efficiency. Empirically, BHyT demonstrates improved stability and efficiency during pretraining, achieving an average of 15.8% faster training and an average of 4.2% higher token generation throughput compared to RMSNorm., while matching or surpassing its inference performance and robustness across language understanding and reasoning benchmarks. Our code is available at: this https URL
Abstract (translated)
预层归一化（Pre-LN）是大规模语言模型（LLMs）的默认选择，对于稳定的预训练和有效的迁移学习至关重要。然而，由于重复统计计算的存在，Pre-LN在效率方面表现不佳，并且随着网络深度增加会遇到“深度诅咒”的问题。随着层数的增长，隐藏状态的幅度和方差也会增大，从而导致训练变得不稳定。为了解决这一问题，一些专注于提高速度的方法如动态Tanh（DyT）虽然提高了计算效率，但在深层模型中的稳定性仍然较差。 为了同时解决稳定性和效率的问题，我们提出了有界双曲正切函数(Bounded Hyperbolic Tanh, BHyT)，这是一种可直接替换Pre-LN的方案。BHyT通过结合tanh非线性函数和基于数据驱动的显式输入边界来确保激活值保持在非饱和范围内，从而防止了随着深度增加而产生的幅度和方差增长，并提供了一个理论上的稳定性保证。 为了提高效率，BHyT每个块只计算一次精确统计量，并用一个轻量级的方差近似替换第二个归一化步骤，以进一步提升效率。实验证明，在预训练过程中，与RMSNorm相比，BHyT在保持或超越其推理性能和健壮性的同时，实现了平均15.8%更快的训练速度以及4.2%更高的标记生成吞吐量。 我们的代码可以在[此处](this https URL)获取。
URL
https://arxiv.org/abs/2601.09719
