Abstract
While supervised fine-tuning (SFT) has been a straightforward approach for tailoring the output of foundation large language model (LLM) to specific preferences, concerns have been raised about the depth of this alignment, with some critiques suggesting it is merely "superficial". We critically examine this hypothesis within the scope of cross-lingual generation tasks, proposing that the effectiveness of SFT may be constrained by its reliance on prior tokens to guide cross-lingual generation. Based on this crucial insight, and in response to the challenges posed by the costly and limited availability of non-English data for SFT, we introduce a novel training-free alignment method named PreTTY, which employs minimal task-related prior tokens to bridge the foundation LLM and the SFT LLM, achieving comparable performance without training. Experiments on machine translation and part-of-speech tagging across eight languages demonstrate the efficacy of PreTTY in cross-lingual settings. Remarkably, by initiating the decoding process with only one or two prior tokens, foundation LLMs can achieve performance comparable to their SFT counterparts. This method presents a cost-effective alternative to SFT and advances the democratization of multilingual LLMs.
Abstract (translated)
虽然监督微调（SFT）已经是一种将基础大型语言模型（LLM）输出定制到特定偏好的直接方法，但人们对其深度的担忧也随之提出，有些批评认为这仅仅是“表面”。我们将在跨语言生成任务的范围内对这一假设进行深入探讨，并提出一个名为PreTTY的新训练-免费对齐方法，该方法采用最小化任务相关的前缀来连接基础LLM和SFT LLM，实现与训练相同或更好的性能，而无需训练。在八种语言的机器翻译和词性标注实验中，证明了PreTTY在跨语言环境中的有效性。值得注意的是，通过仅使用前几个前缀启动解码过程，基础LLM可以实现与SFT同级的性能。这种方法为SFT提供了一种成本效益高的替代方案，并推动了多语言LLM的民主化。
URL
https://arxiv.org/abs/2404.16766
