Abstract
Large Language Models (LLMs) have exhibited remarkable performance across various natural language processing (NLP) tasks. However, fine-tuning these models often necessitates substantial supervision, which can be expensive and time-consuming to obtain. This paper introduces a novel unsupervised method called LanguageModel Self-Improvement by Reinforcement Learning Contemplation (SIRLC) that improves LLMs without reliance on external labels. Our approach is grounded in the observation that it is simpler for language models to assess text quality than to generate text. Building on this insight, SIRLC assigns LLMs dual roles as both student and teacher. As a student, the LLM generates answers to unlabeled questions, while as a teacher, it evaluates the generated text and assigns scores accordingly. The model parameters are updated using reinforcement learning to maximize the evaluation score. We demonstrate that SIRLC can be applied to various NLP tasks, such as reasoning problems, text generation, and machine translation. Our experiments show that SIRLC effectively improves LLM performance without external supervision, resulting in a 5.6% increase in answering accuracy for reasoning tasks and a rise in BERTScore from 0.82 to 0.86 for translation tasks. Furthermore, SIRLC can be applied to models of different sizes, showcasing its broad applicability.
Abstract (translated)
大型语言模型(LLM)在多种自然语言处理任务(NLP任务)中表现出卓越的性能。然而，精细调整这些模型通常需要充分的监督，这可能会变得非常昂贵和耗时。本文介绍了一种名为“语言模型自我强化学习协商(SIRLC)”的全新的无监督方法，该方法无需外部标签来提高LLM的性能。我们的研究方法基于观察，即语言模型评估文本质量比生成文本更简单。基于这一洞察力，SIRLC将LLM赋予双重角色，既是学生也是教师。作为学生，LLM生成未标记的问题答案，作为教师，它评估生成的文本并相应地分配分数。模型参数通过强化学习更新以最大化评估分数。我们证明了SIRLC可以应用于各种NLP任务，例如推理问题、文本生成和机器翻译。我们的实验表明，SIRLC在没有外部监督的情况下有效地提高了LLM的性能，推理任务回答准确率的增加率为5.6%，翻译任务BERTScore的提升率为0.82到0.86。此外，SIRLC可以应用于不同大小的语言模型，展示其广泛的适用性。
URL
https://arxiv.org/abs/2305.14483
