Abstract
Quantization and fine-tuning are crucial for deploying large language models (LLMs) on resource-constrained edge devices. However, fine-tuning quantized models presents significant challenges, primarily stemming from: First, the mismatch in data types between the low-precision quantized weights (e.g., 4-bit) and the high-precision adaptation weights (e.g., 16-bit). This mismatch limits the computational efficiency advantage offered by quantized weights during inference. Second, potential accuracy degradation when merging these high-precision adaptation weights into the low-precision quantized weights, as the adaptation weights often necessitate approximation or truncation. Third, as far as we know, no existing methods support the lossless merging of adaptation while adjusting all quantized weights. To address these challenges, we introduce lossless ternary adaptation for quantization-aware fine-tuning (LoTA-QAF). This is a novel fine-tuning method specifically designed for quantized LLMs, enabling the lossless merging of ternary adaptation weights into quantized weights and the adjustment of all quantized weights. LoTA-QAF operates through a combination of: i) A custom-designed ternary adaptation (TA) that aligns ternary weights with the quantization grid and uses these ternary weights to adjust quantized weights. ii) A TA-based mechanism that enables the lossless merging of adaptation weights. iii) Ternary signed gradient descent (t-SignSGD) for updating the TA weights. We apply LoTA-QAF to Llama-3.1/3.3 and Qwen-2.5 model families and validate its effectiveness on several downstream tasks. On the MMLU benchmark, our method effectively recovers performance for quantized models, surpassing 16-bit LoRA by up to 5.14\%. For task-specific fine-tuning, 16-bit LoRA achieves superior results, but LoTA-QAF still outperforms other methods.
Abstract (translated)
量化和微调对于在资源受限的边缘设备上部署大型语言模型（LLMs）至关重要。然而，对量化模型进行微调带来了显著的挑战，主要原因是： 首先，低精度量化权重（例如4位）与高精度适应权重（例如16位）之间存在数据类型不匹配的问题。这种不匹配限制了在推理过程中量化权重提供的计算效率优势。 其次，在将这些高精度适应权重合并到低精度量化权重时可能出现准确度下降的情况，因为适应权重往往需要近似或截断处理。 第三，据我们所知，目前尚无任何方法支持在调整所有量化权重的同时进行无损融合适应过程。 为解决上述挑战，我们提出了一种面向量化感知微调（QAT）的无损三值适应技术（LoTA-QAF）。这是一种专门针对量化LLMs设计的新颖微调方法，它能够实现将三值适应权重无损地合并到量化权重中，并调整所有量化权重。LoTA-QAF通过以下组合方式运行： i) 设计了一种自定义的三值适应（TA），该机制使三值权重与量化网格对齐，并使用这些三值权重来调整量化权重。 ii) 基于TA的机制，使得可以无损地融合适应权重。 iii) 用于更新TA权重的三值有符号梯度下降法（t-SignSGD）。 我们将LoTA-QAF应用于Llama-3.1/3.3和Qwen-2.5模型家族，并在多个下游任务上验证了其有效性。在MMLU基准测试中，我们的方法有效恢复了量化模型的性能，比16位低秩适应（LoRA）高出最多5.14%。对于特定任务的微调而言，虽然16位LoRA表现出色，但LoTA-QAF仍然优于其他方法。 这种方法在保持计算效率的同时提高了量化LLMs的精度和灵活性，在资源受限设备上的应用前景广阔。
URL
https://arxiv.org/abs/2505.18724
