Abstract
What is the most brute-force way to install interpretable, controllable features into a model's activations? Controlling how LLMs internally represent concepts typically requires sophisticated methods to first identify, then intervene on the model's existing feature geometry. We bypass all of this. We finetune an LLM with a simple auxiliary loss, training 16 of its 3072 residual stream dimensions to be inert interpretability flags that simply indicate what concepts are required for generation. The model reorganizes around them anyway, learning to rely on these flags during actual generation tasks. As a result, these inert flags become genuine internal features: interpretable control switches that allow us to steer generation at inference time. Why does this work? When a feature is reliably supplied at a fixed location, gradient descent gradually eliminates redundant encodings elsewhere, and the model erodes its own alternative representations. A model's efficiency pressure is a lever - exploitable to induce interpretable, controllable representations.
Abstract (translated)
最直接的强行方式是在模型激活中安装可解释和可控的功能是什么？控制LLM（大型语言模型）内部的概念表示通常需要先通过复杂的方法来识别，然后干预模型现有的特征几何结构。我们绕过了这一切。 我们通过对LLM进行微调，并添加一个简单的辅助损失函数，训练其3072个残差流维度中的16个维度成为惰性的可解释标志，这些标志简单地指示生成所需的概念。尽管如此，模型还是围绕这些标志进行了重组，并学会了在实际生成任务中依赖这些标志。 结果是，这些原本的惰性标志变成了真正的内部特征：允许我们在推理时间进行引导的可解释控制开关。 为什么这种方法有效？当一个功能在一个固定位置可靠地提供时，梯度下降会逐渐消除其他地方冗余的编码，模型也会侵蚀其自身的替代表示。模型的效率压力是一个杠杆，可以被利用来诱导出可解释且可控的表示形式。
URL
https://arxiv.org/abs/2602.07828
