Abstract
While Large Language Models (LLMs) have demonstrated proficiency in handling complex queries, much of the past work has depended on extensively annotated datasets by human experts. However, this reliance on fully-supervised annotations poses scalability challenges, particularly as models and data requirements grow. To mitigate this, we explore the potential of enhancing LLMs' reasoning abilities with minimal human supervision. In this work, we introduce self-reinforcement, which begins with Supervised Fine-Tuning (SFT) of the model using a small collection of annotated questions. Then it iteratively improves LLMs by learning from the differences in responses from the SFT and unfinetuned models on unlabeled questions. Our approach provides an efficient approach without relying heavily on extensive human-annotated explanations. However, current reasoning benchmarks typically only include golden-reference answers or rationales. Therefore, we present \textsc{PuzzleBen}, a weakly supervised benchmark that comprises 25,147 complex questions, answers, and human-generated rationales across various domains, such as brainteasers, puzzles, riddles, parajumbles, and critical reasoning tasks. A unique aspect of our dataset is the inclusion of 10,000 unannotated questions, enabling us to explore utilizing fewer supersized data to boost LLMs' inference capabilities. Our experiments underscore the significance of \textsc{PuzzleBen}, as well as the effectiveness of our methodology as a promising direction in future endeavors. Our dataset and code will be published soon on \texttt{Anonymity Link}.
Abstract (translated)
大规模语言模型（LLMs）在处理复杂查询方面表现出熟练程度，但很大程度上过去的工作都依赖于人类专家充分标注的 datasets。然而，对完全监督标注的依赖在模型和数据要求增长时提出了可扩展性挑战。为了减轻这一依赖，我们探讨了通过最小限度的人类监督增强LLM推理能力的前景。在这项工作中，我们引入了自监督强化，从使用一小部分已标注的问题对模型进行监督微调开始。然后它通过学习来自监督和未微调模型的回答差异来逐步改进LLM。我们的方法提供了一种高效的方法，没有依赖大量的人类标注解释。然而，当前的推理基准通常仅包括黄金参考答案或理由。因此，我们提出了 \textsc{PuzzleBen}，一个弱监督基准，它包括了各种领域的25,147个复杂问题、答案和人类生成的推理。我们数据集中的一个独特之处是包括了10,000个未标注的问题，使我们能够探索使用更少的超参数数据来提高LLM的推理能力。我们的实验强调了 \textsc{PuzzleBen} 的意义，以及我们方法作为未来探索的一个有前途的方向的重要性。我们的数据和代码很快将发表在 \texttt{Anonymity Link} 上。
URL
https://arxiv.org/abs/2405.04086
