Abstract
Activation steering methods enable inference-time control of large language model (LLM) behavior without retraining, but current approaches face a fundamental trade-off: sample-efficient methods suboptimally capture steering signals from labeled examples, while methods that better extract these signals require hundreds to thousands of examples. We introduce COLD-Steer, a training-free framework that steers LLM activations by approximating the representational changes that would result from gradient descent on in-context examples. Our key insight is that the effect of fine-tuning on a small set of examples can be efficiently approximated at inference time without actual parameter updates. We formalize this through two complementary approaches: (i) a unit kernel approximation method that updates the activations directly using gradients with respect to them, normalized across examples, and (ii) a finite-difference approximation requiring only two forward passes regardless of example count. Experiments across a variety of steering tasks and benchmarks demonstrate that COLD-Steer achieves upto 95% steering effectiveness while using 50 times fewer samples compared to the best baseline. COLD-Steer facilitates accommodating diverse perspectives without extensive demonstration data, which we validate through our experiments on pluralistic alignment tasks. Our framework opens new possibilities for adaptive, context-aware model control that can flexibly address varying loss-driven human preferences through principled approximation of learning dynamics rather than specialized training procedures.
Abstract (translated)
激活引导方法能够在不重新训练的情况下控制大规模语言模型（LLM）的行为，但当前的方法面临一个根本性的权衡：样本效率高的方法无法充分捕捉标记示例中的引导信号，而能够更好地提取这些信号的方法则需要数百到数千个样本。我们提出了COLD-Steer框架，这是一个无需训练的框架，通过近似在上下文示例上进行梯度下降所产生的表示变化来引导LLM激活状态。我们的关键见解在于，在推理时可以通过不实际更新参数的方式高效地模拟针对少量示例微调的效果。 为了实现这一点，我们提出了两种互补的方法：（i）一种单位核近似方法，这种方法直接使用关于这些激活的梯度进行更新，并且在各个样本之间进行了归一化处理；（ii）有限差分近似法仅需要两次前向传递，无论示例数量多少。一系列针对不同引导任务和基准测试的实验表明，COLD-Steer能够在使用比最佳基线少50倍样本的情况下实现高达95%的引导有效性。 通过我们在多样视角一致性的任务上的实验验证了COLD-Steer框架能够促进在无需大量示例数据的情况下适应多元观点。我们的框架为灵活地应对由损失驱动的人类偏好提供了新的可能性，它通过合理近似学习动态而不是专门训练程序来实现这一点，从而开启了自适应的、上下文感知模型控制的新途径。
URL
https://arxiv.org/abs/2603.06495
