Abstract
Effective data selection is critical for efficient training of modern Large Language Models (LLMs). This paper introduces Influence Distillation, a novel, mathematically-justified framework for data selection that employs second-order information to optimally weight training samples. By distilling each sample's influence on a target distribution, our method assigns model-specific weights that are used to select training data for LLM fine-tuning, guiding it toward strong performance on the target domain. We derive these optimal weights for both Gradient Descent and Adam optimizers. To ensure scalability and reduce computational cost, we propose a $\textit{landmark-based approximation}$: influence is precisely computed for a small subset of "landmark" samples and then efficiently propagated to all other samples to determine their weights. We validate Influence Distillation by applying it to instruction tuning on the Tulu V2 dataset, targeting a range of tasks including GSM8k, SQuAD, and MMLU, across several models from the Llama and Qwen families. Experiments show that Influence Distillation matches or outperforms state-of-the-art performance while achieving up to $3.5\times$ faster selection.
Abstract (translated)
有效数据选择对于现代大型语言模型（LLM）的高效训练至关重要。本文介绍了影响蒸馏（Influence Distillation），这是一种新颖且数学上合理化的数据选择框架，利用二阶信息来优化加权训练样本的方式。通过提炼每个样本对目标分布的影响，我们的方法为LLM微调分配特定于模型的权重，从而指导其在目标领域的性能表现。我们推导了这两种最常用优化器（梯度下降和Adam）下的最优权重。 为了确保可扩展性和降低计算成本，我们提出了一种基于地标样例的近似方法：精确地为一小部分“地标”样本计算影响值，并然后高效传播到所有其他样本上以确定其权重。我们在Tulu V2数据集上的指令调优任务中验证了影响蒸馏的有效性，针对包括GSM8k、SQuAD和MMLU等不同任务类型，并覆盖来自Llama和Qwen家族的多个模型。 实验结果表明，通过影响蒸馏选择训练数据能够与现有最佳方法相匹敌或超越其性能表现，同时实现高达3.5倍更快的选择速度。
URL
https://arxiv.org/abs/2505.19051
