Abstract
The scaling of Generative AI (GenAI) models into the hundreds of billions of parameters makes low-precision computation indispensable for efficient deployment. We argue that the fundamental solution lies in developing low-precision floating-point formats, which inherently provide numerical stability, memory savings, and hardware efficiency without dequantization overhead. In this paper, we present a theoretical and empirical study of an exponent concentration phenomenon in GenAI weights: exponents consistently exhibit low entropy across architectures and modalities. We show that this arises naturally from $\alpha$-stable distributions induced by stochastic gradient descent, and we prove tight bounds on the entropy of exponents. Our analysis establishes a theoretical compression limit near FP4.67, which motivates the design of a practical FP8 format. Building on these insights, we propose Exponent-Concentrated FP8 (ECF8), a lossless compression framework with entropy-aware encoding and GPU-optimized decoding. Experiments on LLMs and DiTs up to 671B parameters demonstrate up to 26.9% memory savings and 177.1% throughput acceleration, with perfectly lossless computations, i.e., no deviation in model outputs. Our results establish exponent concentration as a statistical law of trained models and open a principled path for lossless low-precision floating-point design in the FP8 era.
Abstract (translated)
生成式AI（GenAI）模型的参数量扩展到数百亿级别，使得低精度计算成为高效部署不可或缺的一部分。我们认为，根本性的解决方案在于开发提供数值稳定性、内存节省和硬件效率的低精度浮点格式，并且无需去量化开销。在本文中，我们提出了关于GenAI权重中的指数集中现象的一个理论与实证研究：无论架构还是模式如何，指数部分始终表现出较低的熵值。我们展示了这种现象自然地由随机梯度下降所诱导的α-稳定分布产生，并证明了指数熵值得紧致界。我们的分析建立了接近FP4.67的理论压缩极限，从而激励设计一个实用的FP8格式。基于这些见解，我们提出了指数集中型FP8（ECF8），这是一个无损压缩框架，具备熵感知编码和GPU优化解码功能。在高达671B参数量的LLM和DiT上的实验表明，最多可节省26.9%的内存，并且性能加速可达177.1%，同时确保完全无损计算，即模型输出没有偏差变化。我们的结果确立了指数集中作为训练模型中的统计规律的地位，并为FP8时代的无损低精度浮点设计开辟了一条有原则的道路。
URL
https://arxiv.org/abs/2510.02676
