Abstract
Recently, training-free methods for improving large language models (LLMs) have attracted growing interest, with token-level attention tuning emerging as a promising and interpretable direction. However, existing methods typically rely on auxiliary mechanisms to identify important or irrelevant task-specific tokens, introducing potential bias and limiting applicability. In this paper, we uncover a surprising and elegant alternative: the semantically empty initial token is a powerful and underexplored control point for optimizing model behavior. Through theoretical analysis, we show that tuning the initial token's attention sharpens or flattens the attention distribution over subsequent tokens, and its role as an attention sink amplifies this effect. Empirically, we find that: (1) tuning its attention improves LLM performance more effectively than tuning other task-specific tokens; (2) the effect follows a consistent trend across layers, with earlier layers having greater impact, but varies across attention heads, with different heads showing distinct preferences in how they attend to this token. Based on these findings, we propose ZeroTuning, a training-free approach that improves LLM performance by applying head-specific attention adjustments to this special token. Despite tuning only one token, ZeroTuning achieves higher performance on text classification, multiple-choice, and multi-turn conversation tasks across models such as Llama, Qwen, and DeepSeek. For example, ZeroTuning improves Llama-3.1-8B by 11.71% on classification, 2.64% on QA tasks, and raises its multi-turn score from 7.804 to 7.966. The method is also robust to limited resources, few-shot settings, long contexts, quantization, decoding strategies, and prompt variations. Our work sheds light on a previously overlooked control point in LLMs, offering new insights into both inference-time tuning and model interpretability.
Abstract (translated)
最近，无需训练的方法在提升大型语言模型（LLMs）性能方面引起了越来越多的关注，其中基于令牌级别的注意力微调展现出了前景和可解释性。然而，现有的方法通常依赖辅助机制来识别特定任务的重要或不相关令牌，这可能会引入潜在的偏差并限制其适用范围。在这篇论文中，我们揭示了一个令人惊讶且优雅的替代方案：语义空缺的初始标记是一个强大的且被低估了的控制点，可用于优化模型行为。通过理论分析，我们展示了调整初始标记的注意力可以锐化或扁平化后续令牌上的注意力分布，并且作为注意力汇流的角色会放大这种效果。从实验上来看，我们发现： 1. 调整其注意力比调整个别任务特定令牌更有效地提高LLM性能； 2. 效果在各层中呈现出一致的趋势，早期层影响更大，但不同注意头的效果有所不同，显示出它们如何关注这个标记的偏好各异。 基于这些发现，我们提出了ZeroTuning——一种无需训练的方法，通过针对该特殊标记进行头部特定的注意力调整来提升LLM性能。尽管只调一个令牌，但ZeroTuning在Llama、Qwen和DeepSeek等模型上，在文本分类、多项选择以及多轮对话任务上的表现均超越了现有方法。例如，ZeroTuning使Llama-3.1-8B在分类任务中的性能提升了11.71%，在QA任务中提高了2.64%，并且将其多轮对话得分从7.804提升到了7.966。此外，该方法对于资源有限、少量样本设置、长上下文、量化和解码策略以及提示变化等场景也表现出较强的鲁棒性。 我们的工作揭示了LLMs中一个此前被忽略的控制点，并为推理时调整和模型可解释性提供了新的见解。
URL
https://arxiv.org/abs/2505.11739
