Abstract
Large language Models (LLMs) are highly sensitive to variations in prompt formulation, which can significantly impact their ability to generate accurate responses. In this paper, we introduce a new task, Prompt Sensitivity Prediction, and a dataset PromptSET designed to investigate the effects of slight prompt variations on LLM performance. Using TriviaQA and HotpotQA datasets as the foundation of our work, we generate prompt variations and evaluate their effectiveness across multiple LLMs. We benchmark the prompt sensitivity prediction task employing state-of-the-art methods from related tasks, including LLM-based self-evaluation, text classification, and query performance prediction techniques. Our findings reveal that existing methods struggle to effectively address prompt sensitivity prediction, underscoring the need to understand how information needs should be phrased for accurate LLM responses.
Abstract (translated)
大型语言模型（LLMs）对提示语的细微变化非常敏感，这会显著影响它们生成准确回复的能力。本文介绍了新的任务——提示语敏感性预测（Prompt Sensitivity Prediction），以及一个名为PromptSET的数据集，用于研究轻微的提示语变化如何影响LLM的表现。基于TriviaQA和HotpotQA数据集，我们生成了不同的提示语变体，并对多个大型语言模型进行了有效性评估。我们使用来自相关任务的前沿方法来基准测试提示敏感性预测任务，包括基于LLM的自我评价、文本分类以及查询性能预测技术等。研究发现表明，现有的方法难以有效解决提示敏感性预测问题，这凸显了理解信息需求如何表述以获得准确的LLM回复的重要性。
URL
https://arxiv.org/abs/2502.06065
