Abstract
Progress in Natural Language Processing (NLP) has been dictated by the rule of more: more data, more computing power and more complexity, best exemplified by the Large Language Models. However, training (or fine-tuning) large dense models for specific applications usually requires significant amounts of computing resources. This \textbf{Ph.D. dissertation} focuses on an under-investi\-gated NLP data engineering technique, whose potential is enormous in the current scenario known as Instance Selection (IS). The IS goal is to reduce the training set size by removing noisy or redundant instances while maintaining the effectiveness of the trained models and reducing the training process cost. We provide a comprehensive and scientifically sound comparison of IS methods applied to an essential NLP task -- Automatic Text Classification (ATC), considering several classification solutions and many datasets. Our findings reveal a significant untapped potential for IS solutions. We also propose two novel IS solutions that are noise-oriented and redundancy-aware, specifically designed for large datasets and transformer architectures. Our final solution achieved an average reduction of 41\% in training sets, while maintaining the same levels of effectiveness in all datasets. Importantly, our solutions demonstrated speedup improvements of 1.67x (up to 2.46x), making them scalable for datasets with hundreds of thousands of documents.
Abstract (translated)
自然语言处理（NLP）的进步一直遵循“越多越好”的原则：更多的数据、更强的计算能力和更复杂的模型，这一点在大型语言模型中得到了最佳体现。然而，为特定应用训练（或微调）大而密集的模型通常需要大量的计算资源。本**博士论文**重点研究了一种未被充分探索的NLP数据工程技术——实例选择（IS），其在当前背景下具有巨大的潜力。IS的目标是通过移除噪声或冗余样本减少训练集大小，同时保持训练出的模型的有效性并降低训练成本。 我们在几种分类解决方案和许多数据集上对IS方法进行了全面而科学的比较，特别是针对NLP中的关键任务——自动文本分类（ATC）。我们的研究发现揭示了IS解决方案的巨大未开发潜力。此外，我们还提出了两种新的噪声导向和冗余感知的IS方案，专门设计用于大规模数据集和Transformer架构。 最终，我们的解决方案在所有数据集中平均减少了41%的训练集大小，同时保持相同的有效性水平。重要的是，我们的方法展示了高达1.67倍（最高达2.46倍）的速度提升，使其能够扩展到包含数十万文档的数据集上。
URL
https://arxiv.org/abs/2506.07169
