Abstract
Data selection plays a crucial role in data-driven decision-making, including in large language models (LLMs), and is typically task-dependent. Properties such as data quality and diversity have been extensively studied and are known to enhance model performance. However, it remains unclear whether there exist other quantitative and general principles of data selection that can consistently improve performance, especially for complex tasks with limited prior knowledge. In this paper, we demonstrate that selecting more uniformly distributed data can improve training efficiency while enhancing performance. Specifically, we establish that more uniform (less biased) distribution leads to a larger minimum pairwise distance between data points, denoted by $h_{\min}$, and prove that a smaller $h_{\min}$ can slow down the training dynamics of gradient descent (GD). Moreover, we theoretically show that the approximation error of neural networks decreases as $h_{\min}$ increases. Our analysis introduces a convergence framework for GD beyond the Neural Tangent Kernel (NTK) regime, applicable to a broad class of architectures, including transformers, without requiring Lipschitz smoothness. This framework further provides theoretical justification for the use of residual connections and function compositions in deep neural architectures. In the end, we conduct comprehensive experiments for supervised fine-tuning across various settings, including different optimization strategies, model sizes, and training datasets. The results consistently demonstrate that selecting data by maximizing pairwise distance significantly accelerates training and achieves comparable or better performance in LLMs across diverse datasets. Code and Datasets are available at the link: this https URL.
Abstract (translated)
数据选择在基于数据的决策制定中扮演着关键角色，特别是在大型语言模型（LLMs）中，并且通常是任务相关的。诸如数据质量和多样性的属性已经被广泛研究并且被证明可以提高模型性能。然而，是否存在其他能够一致提升性能的数据选择定量和通用原则，尤其是对于那些先验知识有限的复杂任务来说，这一点尚不清楚。 在本文中，我们展示了选择更均匀分布的数据可以在提高训练效率的同时增强性能。具体而言，我们确立了数据点之间最小成对距离（记为 $h_{\min}$）随分布更加均匀而增加，并证明较小的 $h_{\min}$ 可以减缓梯度下降 (GD) 的训练动态。此外，理论上显示神经网络的近似误差随着 $h_{\min}$ 的增大而减少。 我们的分析引入了一个超越神经切片核（NTK）框架的收敛性理论框架，适用于包括变压器在内的广泛架构类别，并且不需要利普希茨平滑度。此框架进一步提供了残差连接和函数组合在深度神经网络结构中的使用依据。 最终，我们在各种设置下进行了全面的监督微调实验，包括不同的优化策略、模型大小和训练数据集。结果一致表明，在大型语言模型中，通过最大化成对距离来选择数据可以显著加速训练，并且在不同数据集上可获得相当或更好的性能。 代码和数据集可以在以下链接获取：[此URL]（请将 [此URL] 替换为实际的链接地址）。
URL
https://arxiv.org/abs/2506.24120
