Abstract
Zero-shot text classification typically relies on prompt engineering, but the inherent prompt brittleness of large language models undermines its reliability. Minor changes in prompt can cause significant discrepancies in model performance. We attribute this prompt brittleness largely to the narrow focus on nexttoken probabilities in existing methods. To address this, we propose Placeholding Parallel Prediction (P3), a novel approach that predicts token probabilities across multiple positions and simulates comprehensive sampling of generation paths in a single run of a language model. Experiments show improved accuracy and up to 98% reduction in the standard deviation across prompts, boosting robustness. Even without a prompt, P3 maintains comparable performance, reducing the need for prompt engineering.
Abstract (translated)
零样本文本分类通常依赖于提示工程，但大型语言模型内在的提示脆弱性影响了其可靠性。即使是微小的提示变化也可能导致模型性能出现显著差异。我们认为这种提示脆弱性的主要原因是现有方法对下一个词概率的狭窄关注。为解决这一问题，我们提出了一种新颖的方法——占位并行预测（Placeholding Parallel Prediction, P3），该方法可以同时在多个位置预测词的概率，并在一个语言模型运行过程中模拟生成路径的全面采样。实验表明，P3 方法不仅提高了准确性，还使提示间的标准差降低了高达 98%，从而增强了模型的稳健性。即使没有提示的情况下，P3 方法也能保持相当的性能水平，减少了对复杂提示工程的需求。
URL
https://arxiv.org/abs/2504.03159
