Abstract
The Muon optimizer, a matrix-structured algorithm that leverages spectral orthogonalization of gradients, is a milestone in the pretraining of large language models. However, the underlying mechanisms of Muon -- particularly the role of gradient orthogonalization -- remain poorly understood, with very few works providing end-to-end analyses that rigorously explain its advantages in concrete applications. We take a step by studying the effectiveness of a simplified variant of Muon through two case studies: matrix factorization, and in-context learning of linear transformers. For both problems, we prove that simplified Muon converges linearly with iteration complexities independent of the relevant condition number, provably outperforming gradient descent and Adam. Our analysis reveals that the Muon dynamics decouple into a collection of independent scalar sequences in the spectral domain, each exhibiting similar convergence behavior. Our theory formalizes the preconditioning effect induced by spectral orthogonalization, offering insight into Muon's effectiveness in these matrix optimization problems and potentially beyond.
Abstract (translated)
穆安（Muon）优化器是一种基于矩阵结构的算法，利用梯度谱正交化技术，在大规模语言模型预训练中取得了重要进展。然而，关于穆安的具体工作机制——特别是梯度正交化的角色——仍然知之甚少，鲜有研究能提供严格的端到端分析来详细解释其在实际应用中的优势。 为了更好地理解这一点，我们通过两个案例研究简化版的穆安优化器的有效性：矩阵分解和线性变换器的上下文学习。对于这两个问题，我们证明了简化的穆安算法能够以与相关条件数无关的迭代复杂度进行线性收敛，并且在理论上可以优于梯度下降法和Adam优化器。 我们的分析揭示出，当进入频谱域时，穆安动力学会分解为一系列独立的标量序列，每个序列都表现出相似的收敛行为。这一理论形式化了由谱正交化引起的预处理效果，不仅解释了在矩阵优化问题中穆安的有效性，还可能提供对其它领域的见解和应用潜力。
URL
https://arxiv.org/abs/2601.13474
