Abstract
Precise attribute intensity control--generating Large Language Model (LLM) outputs with specific, user-defined attribute intensities--is crucial for AI systems adaptable to diverse user expectations. Current LLM alignment methods, however, typically provide only directional or open-ended guidance, failing to reliably achieve exact attribute intensities. We address this limitation with three key designs: (1) reformulating precise attribute intensity control as a target-reaching problem, rather than simple maximization; (2) training a lightweight value function via temporal-difference learning to predict final attribute intensity scores from partial generations, thereby steering LLM outputs; and (3) employing gradient-based interventions on hidden representations to navigate the model precisely towards specific attribute intensity targets. Our method enables fine-grained, continuous control over attribute intensities, moving beyond simple directional alignment. Experiments on LLaMA-3.2-3b and Phi-4-mini confirm our method's ability to steer text generation to user-specified attribute intensities with high accuracy. Finally, we demonstrate efficiency enhancements across three downstream tasks: preference data synthesis, Pareto frontier approximation and optimization, and distillation of aligned behaviors for intervention-free inference. Our code is available on this https URL
Abstract (translated)
精确的属性强度控制——即根据用户定义的具体强度生成大型语言模型（LLM）输出——对于适应多样化用户期望的人工智能系统至关重要。然而，当前的语言模型对齐方法通常只能提供方向性或开放式的指导，并不能可靠地实现确切的属性强度。为了解决这一局限性，我们提出了三个关键设计：(1) 将精确属性强度控制重新定义为目标导向问题而非简单最大化；(2) 通过时差学习训练一个轻量级的价值函数来预测从部分生成中得到的最终属性强度分数，并据此引导LLM输出；以及 (3) 使用基于梯度的方法干预隐藏表示，以使模型准确地导航到特定的属性强度目标。我们的方法能够实现细粒度、连续控制的属性强度调整，超越了简单的方向对齐。在LLaMA-3.2-3b和Phi-4-mini上的实验确认了我们的方法能够将文本生成引导至用户指定的具体属性强度上，并且精度很高。最后，我们展示了该方法在三个下游任务中的效率提升：偏好数据合成、帕累托前沿逼近与优化以及无干预推理的对齐行为精炼。我们的代码可在[此链接](https://这个位置填写实际的URL)获取。
URL
https://arxiv.org/abs/2510.12121
