Abstract
Large language models can produce toxic or inappropriate text even for benign inputs, creating risks when deployed at scale. Detoxification is therefore important for safety and user trust, particularly when we want to reduce harmful content without sacrificing the model's generation quality. Many existing approaches rely on model retraining, gradients, or learned auxiliary components, which can be costly and may not transfer across model families or to truly black-box settings. We introduce a test-time procedure that approximates the gradient of completion toxicity with respect to the input embeddings and uses a small number of descent steps to steer generation toward less toxic continuations. This is achieved with zeroth-order optimization that requires only access to input embeddings, a toxicity scoring function, and forward evaluations of the model. Empirically, the approach delivers robust toxicity reductions across models and prompts and, in most settings, achieves the best overall toxicity-quality trade-off. More broadly, our work positions word embeddings as effective control variables and encourages wider use of black-box optimization to guide autoregressive language models toward scalable, safer text generation, without requiring any training or access to intermediate computations.
Abstract (translated)
大型语言模型即使在接收良性输入时，也可能生成有毒或不适当的内容，在大规模部署时会带来风险。因此，为了确保安全性和用户信任，去毒化工作尤为重要，尤其是在减少有害内容的同时又不牺牲模型的生成质量的情况下。现有的许多方法依赖于重新训练、梯度计算或学习辅助组件，这些方法成本较高，并且可能无法跨不同模型家族迁移或应用于真正意义上的黑盒设置中。 我们介绍了一种在测试时进行的过程，该过程通过输入嵌入（input embeddings）来近似完成任务的毒性相对于输入变化的梯度，并使用少量下降步骤将生成引导到更少毒害性的延续。这一方法是基于零阶优化实现的，仅需访问输入嵌入、一个有毒性评分函数和模型的前向评估即可。 从经验来看，在大多数情况下，这种方法能够有效地减少各种模型和提示中的毒性内容，并且在大多数设置下实现了最佳的整体毒性与质量权衡。更为广泛而言，我们的工作将词嵌入作为有效的控制变量进行了定位，并鼓励更广泛应用黑盒优化来引导自回归语言模型实现可扩展、安全的文本生成，而无需任何训练或对中间计算的访问需求。
URL
https://arxiv.org/abs/2602.02498
