Abstract
Large language model alignment is widely used and studied to avoid LLM producing unhelpful and harmful responses. However, the lengthy training process and predefined preference bias hinder adaptation to online diverse human preferences. To this end, this paper proposes an alignment framework, called Reinforcement Learning with Human Behavior (RLHB), to align LLMs by directly leveraging real online human behaviors. By taking the generative adversarial framework, the generator is trained to respond following expected human behavior; while the discriminator tries to verify whether the triplets of query, response, and human behavior come from real online environments. Behavior modeling in natural-language form and the multi-model joint training mechanism enable an active and sustainable online alignment. Experimental results confirm the effectiveness of our proposed methods by both human and automatic evaluations.
Abstract (translated)
大语言模型对齐广泛应用于并受到研究,以避免LLM产生不有用和有害的反应。然而,漫长的训练过程和预定义的偏好偏差会阻碍LLM适应在线多样的人类偏好。为此,本文提出了一种名为强化学习与人类行为(RLHB)的alignment框架,通过直接利用真实在线人类行为来对齐LLM。通过采用生成对抗网络,生成器被训练为根据预期的人类行为响应;而判别器则试图验证三元组查询、响应和人类行为的三元组是否来自真实在线环境。自然语言形式的行为建模和多模型联合训练机制使主动和可持续的在线对齐成为可能。实验结果证实了我们提出方法的既定有效性,同时通过人类和自动评估进行评估。
URL
https://arxiv.org/abs/2405.00578
