Abstract
Interpreting and controlling the internal mechanisms of large language models (LLMs) is crucial for improving their trustworthiness and utility. Recent efforts have primarily focused on identifying and manipulating neurons by establishing discrete mappings between neurons and semantic concepts. However, such mappings struggle to handle the inherent polysemanticity in LLMs, where individual neurons encode multiple, distinct concepts. This makes precise control challenging and complicates downstream interventions. Through an in-depth analysis of both encoder and decoder-based LLMs across multiple text classification datasets, we uncover that while individual neurons encode multiple concepts, their activation magnitudes vary across concepts in distinct, Gaussian-like patterns. Building on this insight, we introduce NeuronLens, a novel range-based interpretation and manipulation framework that provides a finer view of neuron activation distributions to localize concept attribution within a neuron. Extensive empirical evaluations demonstrate that NeuronLens significantly reduces unintended interference, while maintaining precise control for manipulation of targeted concepts, outperforming existing methods.
Abstract (translated)
解释和控制大型语言模型（LLMs）的内部机制对于提高其可靠性和实用性至关重要。近期的研究主要集中在通过建立神经元与语义概念之间的离散映射来识别和操纵神经元上。然而，这种映射方法难以处理大型语言模型中的固有多义性问题，即单个神经元编码多个不同的概念。这使得精确控制变得具有挑战性，并且增加了下游干预的复杂性。 通过对多种文本分类数据集上的基于编码器和解码器的LLMs进行深入分析，我们发现尽管单个神经元编码了多个概念，但它们在不同概念下的激活强度遵循着独特的、类似高斯分布的模式。基于这一洞察，我们引入了NeuronLens——一种新颖的基于范围的解释和操纵框架，该框架提供了对神经元激活分布更细致的视角，以便在单个神经元内定位具体的概念归属。 广泛的实证评估表明，与现有方法相比，NeuronLens显著减少了意外干扰，并保持了对目标概念精确控制的能力。
URL
https://arxiv.org/abs/2502.06809
