Abstract
Recent developments in text classification using Large Language Models (LLMs) in the social sciences suggest that costs can be cut significantly, while performance can sometimes rival existing computational methods. However, with a wide variance in performance in current tests, we move to the question of how to maximize performance. In this paper, we focus on prompt context as a possible avenue for increasing accuracy by systematically varying three aspects of prompt engineering: label descriptions, instructional nudges, and few shot examples. Across two different examples, our tests illustrate that a minimal increase in prompt context yields the highest increase in performance, while further increases in context only tend to yield marginal performance increases thereafter. Alarmingly, increasing prompt context sometimes decreases accuracy. Furthermore, our tests suggest substantial heterogeneity across models, tasks, and batch size, underlining the need for individual validation of each LLM coding task rather than reliance on general rules.
Abstract (translated)
社会科学领域利用大语言模型进行文本分类的最新进展表明，该方法可大幅降低成本，且其性能有时能媲美现有计算方法。然而，由于当前测试中性能表现差异较大，我们转而探讨如何最大化性能。本文聚焦于提示上下文，通过系统性地调整提示工程的三个方面——标签描述、指令引导和少样本示例——来提升准确率。针对两个不同案例的测试显示，提示上下文的微小增加能带来最大的性能提升，而进一步增加上下文仅能带来边际效益。令人担忧的是，增加提示上下文有时反而会降低准确率。此外，我们的测试表明不同模型、任务和批处理规模之间存在显著异质性，这凸显了针对每个大语言模型编码任务进行单独验证的必要性，而非依赖通用规则。
URL
https://arxiv.org/abs/2603.25422
