Abstract
Parameter-efficient fine-tuning (PEFT) large language models (LLMs) have shown impressive performance in various downstream tasks. However, in many real-world scenarios, the collected training data inevitably contains noisy labels. To learn from noisy labels, most solutions select samples with small losses for model training. However, the selected samples, in turn, impact the loss computation in the next iteration. An inaccurate initial selection can create a vicious cycle, leading to suboptimal performance. To break this cycle, we propose Delora, a novel framework that decouples the sample selection from model training. For sample selection, Delora establishes a noisy label detector by introducing clean and noisy LoRA. Benefiting from the memory effect, the clean LoRA is encouraged to memorize clean data, while the noisy LoRA is constrained to memorize mislabeled data, which serves as a learnable threshold for selecting clean and noisy samples. For model training, Delora can use carefully selected samples to fine-tune language models seamlessly. Experimental results on synthetic and real-world noisy datasets demonstrate the effectiveness of Delora in noisy label detection and text classification.
Abstract (translated)
参数高效的微调（PEFT）大型语言模型（LLMs）在各种下游任务中表现出令人印象深刻的性能。然而，在许多现实场景中，收集的训练数据不可避免地包含噪声标签。为从噪声标签中学习，大多数解决方案选择具有小损失值的数据样本进行模型训练。然而，所选样本反过来会影响下一次迭代中的损失计算。初始选择不准确可能会产生恶性循环，导致次优性能。为了打破这个循环，我们提出了 Delora，这是一个将样本选择与模型训练解耦的新型框架。 对于样本选择，Delora 通过引入干净和噪声LoRA建立了噪声标签检测器。得益于记忆效应，干净的 LoRA 被鼓励记住清洁数据，而噪声的 LoRA 则被限制记住错误标记的数据，这作为可学习的阈值用于区分清洁与噪声样本的选择。 对于模型训练，Delora 可以使用精心挑选的样本无缝地微调语言模型。在合成和真实世界的噪声数据集上的实验结果证明了 Delora 在噪声标签检测和文本分类方面的有效性。
URL
https://arxiv.org/abs/2510.10208
