Abstract
Optimization in large language models (LLMs) unfolds over high-dimensional parameter spaces with non-Euclidean structure. Information geometry frames this landscape using the Fisher information metric, enabling more principled learning via natural gradient descent. Though often impractical, this geometric lens clarifies phenomena such as sharp minima, generalization, and observed scaling laws. We argue that curvature-aware approaches deepen our understanding of LLM training. Finally, we speculate on quantum analogies based on the Fubini-Study metric and Quantum Fisher Information, hinting at efficient optimization in quantum-enhanced systems.
Abstract (translated)
在大型语言模型（LLMs）中的优化过程发生在具有非欧几里得结构的高维参数空间中。信息几何学利用费雪信息度量来描绘这一景观，从而使通过自然梯度下降进行更原理性的学习成为可能。尽管这种方法通常难以实现，但它有助于阐明诸如尖锐极小值、泛化和观察到的扩展规律等现象。我们认为，考虑到曲率的方法能加深我们对LLMs训练的理解。最后，基于富比尼-施图迪度量（Fubini-Study metric）和量子费雪信息（Quantum Fisher Information），我们推测出在量子增强系统中的有效优化可能存在的类比关系。
URL
https://arxiv.org/abs/2506.15830
