Abstract
In recent years, the advent of the attention mechanism has significantly advanced the field of natural language processing (NLP), revolutionizing text processing and text generation. This has come about through transformer-based decoder-only architectures, which have become ubiquitous in NLP due to their impressive text processing and generation capabilities. Despite these breakthroughs, language models (LMs) remain susceptible to generating undesired outputs: inappropriate, offensive, or otherwise harmful responses. We will collectively refer to these as ``toxic'' outputs. Although methods like reinforcement learning from human feedback (RLHF) have been developed to align model outputs with human values, these safeguards can often be circumvented through carefully crafted prompts. Therefore, this paper examines the extent to which LLMs generate toxic content when prompted, as well as the linguistic factors -- both lexical and syntactic -- that influence the production of such outputs in generative models.
Abstract (translated)
近年来，注意力机制的出现显著推动了自然语言处理（NLP）领域的发展，彻底革新了文本处理和文本生成。这得益于基于变压器的解码器架构，由于其出色的文本处理和生成能力而成为NLP领域的主流选择。尽管取得了这些突破性进展，语言模型（LMs）仍然容易产生不希望出现的内容：不合适、冒犯性的或具有潜在危害的响应。我们将这些统称为“有毒”输出。虽然诸如从人类反馈中进行强化学习（RLHF）等方法已经开发出来以使模型输出与人类价值观保持一致，但通过精心设计的提示语，这些防护措施常常可以被规避。因此，本文探讨了大型语言模型在接收到特定提示时产生有害内容的程度，以及影响生成此类输出的语言因素——包括词汇和句法方面的因素。
URL
https://arxiv.org/abs/2601.06700
