Abstract
Due to their architecture and vast pre-training data, large language models (LLMs) demonstrate strong text classification performance. However, LLM output - here, the category assigned to a text - depends heavily on the wording of the prompt. While literature on prompt engineering is expanding, few studies focus on classification tasks, and even fewer address domains like psychology, where constructs have precise, theory-driven definitions that may not be well represented in pre-training data. We present an empirical framework for optimizing LLM performance for identifying constructs in texts via prompt engineering. We experimentally evaluate five prompting strategies --codebook-guided empirical prompt selection, automatic prompt engineering, persona prompting, chain-of-thought reasoning, and explanatory prompting - with zero-shot and few-shot classification. We find that persona, chain-of-thought, and explanations do not fully address performance loss accompanying a badly worded prompt. Instead, the most influential features of a prompt are the construct definition, task framing, and, to a lesser extent, the examples provided. Across three constructs and two models, the classifications most aligned with expert judgments resulted from a few-shot prompt combining codebook-guided empirical prompt selection with automatic prompt engineering. Based on our findings, we recommend that researchers generate and evaluate as many prompt variants as feasible, whether human-crafted, automatically generated, or ideally both, and select prompts and examples based on empirical performance in a training dataset, validating the final approach in a holdout set. This procedure offers a practical, systematic, and theory-driven method for optimizing LLM prompts in settings where alignment with expert judgment is critical.
Abstract (translated)
由于其架构和庞大的预训练数据，大型语言模型（LLMs）展现了强大的文本分类性能。然而，LLM的输出——即为文本分配的类别——在很大程度上依赖于提示语的措辞。尽管有关提示工程的研究正在扩展，但很少有研究关注分类任务，并且几乎没有研究针对像心理学这样的领域，在这些领域中，构念具有精确、理论驱动的定义，这可能不会很好地反映在预训练数据中。我们提出了一种通过提示工程优化LLM性能以识别文本中构念的经验框架。我们在零样本和少样本分类中实验性地评估了五种提示策略——代码本引导的经验提示选择、自动提示工程、人格提示、链式思维推理以及解释性提示。我们发现，人格、链式思维和解释并不能完全解决因措辞不佳的提示而导致的表现下降问题。相反，提示中最具有影响力的特征是构念定义、任务框架，并且在一定程度上，提供的示例也起到作用。在三个构念和两个模型中进行测试后，最接近专家判断的分类结果来自一种结合了代码本引导的经验提示选择与自动提示工程的少样本提示。基于我们的研究发现，我们建议研究人员尽可能生成并评估各种各样的提示变体，无论是人工设计、自动生成还是理想情况下同时采用这两种方式，并根据训练数据中的实证表现来选择提示和示例，在保留集中进一步验证最终的方法。这一程序为在需要与专家判断保持一致的环境中优化LLM提示提供了一种实用、系统且理论驱动的方法。
URL
https://arxiv.org/abs/2512.03818
