Abstract
Training and fine-tuning deep learning models, especially large language models (LLMs), on limited and imbalanced datasets poses substantial challenges. These issues often result in poor generalization, where models overfit to dominant classes and underperform on minority classes, leading to biased predictions and reduced robustness in real-world applications. To overcome these challenges, we propose augmenting features in the embedding space by generating synthetic samples using a range of techniques. By upsampling underrepresented classes, this method improves model performance and alleviates data imbalance. We validate the effectiveness of this approach across multiple open-source text classification benchmarks, demonstrating its potential to enhance model robustness and generalization in imbalanced data scenarios.
Abstract (translated)
在有限且不平衡的数据集上训练和微调深度学习模型，尤其是大型语言模型（LLMs），面临着重大挑战。这些问题常常导致模型泛化能力差：过度拟合到主要类别并在次要类别中表现不佳，从而导致预测偏见并降低实际应用中的鲁棒性。为克服这些挑战，我们提出通过在嵌入空间生成合成样本的方法来增强特征，以此来提升未充分代表类别的数据量。这种方法可以提高模型性能，并缓解数据不平衡的问题。我们在多个开源文本分类基准测试中验证了该方法的有效性，展示了其在不均衡数据场景下增强模型鲁棒性和泛化能力的潜力。
URL
https://arxiv.org/abs/2501.06434
