Abstract
Experts in machine learning leverage domain knowledge to navigate decisions in model selection, hyperparameter optimisation, and resource allocation. This is particularly critical for fine-tuning language models (LMs), where repeated trials incur substantial computational overhead and environmental impact. However, no existing automated framework simultaneously tackles the entire model selection and HPO task for resource-efficient LM fine-tuning. We introduce XAutoLM, a meta-learning-augmented AutoML framework that reuses past experiences to optimise discriminative and generative LM fine-tuning pipelines efficiently. XAutoLM learns from stored successes and failures by extracting task- and system-level meta-features to bias its sampling toward fruitful configurations and away from costly dead ends. On four text classification and two question-answering benchmarks, XAutoLM surpasses zero-shot optimiser's peak F1 on five of six tasks, cuts mean evaluation time by up to 4.5x, reduces error ratios by up to sevenfold, and uncovers up to 50% more pipelines above the zero-shot Pareto front. In contrast, simpler memory-based baselines suffer negative transfer. We release XAutoLM and our experience store to catalyse resource-efficient, Green AI fine-tuning in the NLP community.
Abstract (translated)
机器学习专家利用领域知识来指导模型选择、超参数优化和资源配置的决策。特别是在微调语言模型（LMs）时，重复试验会带来巨大的计算开销和环境影响，因此这一过程尤为重要。然而，目前没有现有的自动化框架能够同时处理资源高效的LM微调全过程中的模型选择和超参数优化任务。 我们推出了XAutoLM，这是一个增强型的元学习自动机器学习(AutoML)框架，它通过重用过去的成功与失败经验来高效地优化判别式和生成式语言模型的微调流水线。XAutoLM通过提取任务级和系统级的元特征，从存储的经验中学习，并偏向于采样那些有前途的配置方案，避免无效且成本高昂的努力。 在四个文本分类基准测试和两个问答基准测试上，与零样本优化器相比，XAutoLM在六个任务中的五个任务上实现了最高的F1得分峰值，将平均评估时间缩短了最多4.5倍，减少了错误率高达七倍，并发现多达50%的流水线优于零样本帕累托前沿。 相比之下，较为简单的基于记忆的方法受到负面迁移的影响。我们发布了XAutoLM及其经验存储库，以催化NLP社区中资源高效的绿色AI微调研究和应用。
URL
https://arxiv.org/abs/2508.00924
