Abstract
We introduce XFT, a simple yet powerful training scheme, by simply merging upcycled Mixture-of-Experts (MoE) to unleash the performance limit of instruction-tuned code Large Language Models (LLMs). While vanilla sparse upcycling fails to improve instruction tuning, XFT introduces a shared expert mechanism with a novel routing weight normalization strategy into sparse upcycling, which significantly boosts instruction tuning. After fine-tuning the upcycled MoE model, XFT introduces a learnable model merging mechanism to compile the upcycled MoE model back to a dense model, achieving upcycled MoE-level performance with only dense-model compute. By applying XFT to a 1.3B model, we create a new state-of-the-art tiny code LLM (<3B) with 67.1 and 64.6 pass@1 on HumanEval and HumanEval+ respectively. With the same data and model architecture, XFT improves supervised fine-tuning (SFT) by 13% on HumanEval+, along with consistent improvements from 2% to 13% on MBPP+, MultiPL-E, and DS-1000, demonstrating its generalizability. XFT is fully orthogonal to existing techniques such as Evol-Instruct and OSS-Instruct, opening a new dimension for improving code instruction tuning. Codes are available at this https URL .
Abstract (translated)
我们提出了XFT，一种简单而强大的训练方案，通过简单地将循环利用的MoE与指令微调的代码大语言模型（LLM）合并，释放了LLM的性能限制。尽管经典的稀疏回收方案无法提高指令微调，XFT通过在稀疏回收方案中引入共享专家机制与新颖的路由权重归一化策略，显著提高了指令微调。在微调循环利用的MoE模型后，XFT引入了一个可学习的模型合并机制，将循环利用的MoE模型编译为密歇根模型，实现仅使用密歇根模型计算的循环利用MoE级别性能。将XFT应用于1.3B模型，我们创建了一个新的人工微小代码LLM（<3B），具有HumanEval和HumanEval+上的67.1和64.6分。使用相同的数据和模型架构，XFT在HumanEval+上提高了监督微调（SFT）13%，同时在MBPP+、MultiPL-E和DS-1000上实现了从2%到13%的显著改进，表明其具有良好的泛化能力。与现有的技术如Evol-Instruct和OSS-Instruct等完全正交，XFT为改进代码指令微调打开了一个新的维度。代码可在此https://url.com/处获取。
URL
https://arxiv.org/abs/2404.15247
