Abstract
Large language models (LLMs) are widely used as zero-shot and few-shot classifiers, where task behaviour is largely controlled through prompting. A growing number of works have observed that LLMs are sensitive to prompt variations, with small changes leading to large changes in performance. However, in many cases, the investigation of sensitivity is performed using underspecified prompts that provide minimal task instructions and weakly constrain the model's output space. In this work, we argue that a significant portion of the observed prompt sensitivity can be attributed to prompt underspecification. We systematically study and compare the sensitivity of underspecified prompts and prompts that provide specific instructions. Utilising performance analysis, logit analysis, and linear probing, we find that underspecified prompts exhibit higher performance variance and lower logit values for relevant tokens, while instruction-prompts suffer less from such problems. However, linear probing analysis suggests that the effects of prompt underspecification have only a marginal impact on the internal LLM representations, instead emerging in the final layers. Overall, our findings highlight the need for more rigour when investigating and mitigating prompt sensitivity.
Abstract (translated)
大型语言模型（LLMs）被广泛用作零样本和少量样本分类器，其任务行为主要通过提示来控制。越来越多的研究发现，LLM 对提示的变化非常敏感，即使是微小的改变也可能导致性能上的大幅波动。然而，在许多情况下，对这种敏感性的调查是使用提供最少任务说明且弱化了模型输出空间约束的不明确提示来进行的。在本工作中，我们认为观察到的大部分提示敏感性可以归因于提示的不明确性。我们系统地研究并比较了不明确提示与提供具体指令的提示之间的敏感度差异。通过性能分析、对数几率值分析以及线性探测，我们发现不明确提示表现出更高的性能波动和更低的相关词令牌的对数值，而指令提示则较少受到这些问题的影响。然而，线性探测分析表明，提示不明确性的效果只在模型内部表示的最终层中产生边际影响，而不是影响整体表示结构。总体而言，我们的研究结果强调了在调查和缓解提示敏感性时需要更加严谨的态度。
URL
https://arxiv.org/abs/2602.04297
