Abstract
We explore efficient strategies to fine-tune decoder-only Large Language Models (LLMs) for downstream text classification under resource constraints. Two approaches are investigated: (1) attaching a classification head to a pre-trained causal LLM and fine-tuning on the task (using the LLM's final token embedding as a sequence representation), and (2) instruction-tuning the LLM in a prompt->response format for classification. To enable single-GPU fine-tuning of models up to 8B parameters, we combine 4-bit model quantization with Low-Rank Adaptation (LoRA) for parameter-efficient training. Experiments on two datasets - a proprietary single-label dataset and the public WIPO-Alpha patent dataset (extreme multi-label classification) - show that the embedding-based method significantly outperforms the instruction-tuned method in F1-score, and is very competitive with - even surpassing - fine-tuned domain-specific models (e.g. BERT) on the same tasks. These results demonstrate that directly leveraging the internal representations of causal LLMs, along with efficient fine-tuning techniques, yields impressive classification performance under limited computational resources. We discuss the advantages of each approach while outlining practical guidelines and future directions for optimizing LLM fine-tuning in classification scenarios.
Abstract (translated)
我们探讨了在资源受限条件下，为下游文本分类任务高效微调解码器独享的大规模语言模型（LLMs）的策略。研究调查了两种方法：(1) 将分类头附加到预训练的因果LMM上，并针对特定任务进行微调（使用LMM最终的标记嵌入作为序列表示），以及 (2) 采用指令调整，以提示-响应格式对LLM进行微调用于分类。为了实现单GPU环境下高达80亿参数模型的有效微调，我们结合了4位模型量化与低秩适应（LoRA）技术，以达到高效的训练效果。 在两个数据集上进行了实验：一个是专有的单一标签数据集和公开的WIPO-Alpha专利数据集（极端多标签分类）。实验结果显示，基于嵌入的方法在F1分数方面显著优于指令调优方法，并且与针对特定领域的微调模型（如BERT）相比，在相同的任务中表现出色甚至超越。 这些结果表明，在计算资源有限的情况下，直接利用因果LMM内部表示以及高效的微调技术可以实现卓越的分类性能。我们讨论了每种方法的优势，并提出了优化LLM在分类场景下的微调的实际指南和未来方向。
URL
https://arxiv.org/abs/2512.12677
