Abstract
Post-training quantization (\ptq) had been recently shown as a compromising method to reduce the memory consumption and/or compute cost for large language models. However, a comprehensive study about the effect of different quantization schemes, different model families, different \ptq methods, different quantization bit precision, etc, is still missing. In this work, we provide an extensive study on those components over tens of thousands of zero-shot experiments. Our results show that (1) Fine-grained quantization and \ptq methods (instead of naive round-to-nearest quantization) are necessary to achieve good accuracy and (2) Higher bits (e.g., 5 bits) with coarse-grained quantization is more powerful than lower bits (e.g., 4 bits) with very fine-grained quantization (whose effective bits is similar to 5-bits). We also present recommendations about how to utilize quantization for \llms with different sizes, and leave suggestions of future opportunities and system work that are not resolved in this work.
Abstract (translated)
最近,Post-training quantization (ptq) 被证明是一种降低大型语言模型内存消耗和/或计算成本的妥协方法。然而,关于不同 quantizationScheme、不同模型家族、不同 ptq 方法、不同 quantization bit precision 等的不同效应的全面研究仍然缺失。在本文中,我们对数千次零样本实验中的这些组件进行了广泛的研究。我们的结果显示(1) 精细的量化和 ptq 方法(而不是简单的整数Round-to-nearest量化)是必要的,以实现良好的精度,(2) 粗粒度的量化的更高的位(例如 5 位)比精细的量化的更低的位(例如 4 位)更有威力(其有效位类似于 5 位)。我们还提出了如何对不同大小的 llms 利用量化的建议,并留下了本工作未解决的未来机会和系统工作的建议。
URL
https://arxiv.org/abs/2303.08302
