Abstract
Generative large language models (LLMs) are now used extensively for text classification in computational social science (CSS). In this work, focus on the steps before and after LLM prompting -- conceptualization of concepts to be classified and using LLM predictions in downstream statistical inference -- which we argue have been overlooked in much of LLM-era CSS. We claim LLMs can tempt analysts to skip the conceptualization step, creating conceptualization errors that bias downstream estimates. Using simulations, we show that this conceptualization-induced bias cannot be corrected for solely by increasing LLM accuracy or post-hoc bias correction methods. We conclude by reminding CSS analysts that conceptualization is still a first-order concern in the LLM-era and provide concrete advice on how to pursue low-cost, unbiased, low-variance downstream estimates.
Abstract (translated)
生成式大型语言模型（LLM）现在广泛用于计算社会科学（CSS）中的文本分类。在这项工作中，我们重点关注在LLM提示之前和之后的步骤——即被分类概念的概念化过程以及将LLM预测应用于下游统计推断的过程，我们认为这些环节在大多数使用LLM的时代CSS研究中被忽略了。我们主张，LLM可能会诱使分析人员跳过概念化的步骤，从而产生概念错误，进而导致后续估计结果出现偏差。通过模拟实验，我们展示了仅靠提高LLM的准确性或采用事后偏置校正方法是不足以纠正这种由概念化引起的偏差的。最后，我们提醒CSS分析师，在LLM时代，概念化仍然是一个首要关注点，并提供了如何追求低成本、无偏且方差低的下游估计的具体建议。
URL
https://arxiv.org/abs/2510.03541
