Abstract
The most widely used large language models in the social sciences (such as BERT, and its derivatives, e.g. RoBERTa) have a limitation on the input text length that they can process to produce predictions. This is a particularly pressing issue for some classification tasks, where the aim is to handle long input texts. One such area deals with laws and draft laws (bills), which can have a length of multiple hundred pages and, therefore, are not particularly amenable for processing with models that can only handle e.g. 512 tokens. In this paper, we show results from experiments covering 5 languages with XLM-RoBERTa, Longformer, GPT-3.5, GPT-4 models for the multiclass classification task of the Comparative Agendas Project, which has a codebook of 21 policy topic labels from education to health care. Results show no particular advantage for the Longformer model, pre-trained specifically for the purposes of handling long inputs. The comparison between the GPT variants and the best-performing open model yielded an edge for the latter. An analysis of class-level factors points to the importance of support and substance overlaps between specific categories when it comes to performance on long text inputs.
Abstract (translated)
在社会科学领域中广泛使用的大规模语言模型（如BERT及其衍生版本，例如RoBERTa）在处理生成预测的输入文本长度方面存在限制。对于需要处理长篇输入文本的一些分类任务而言，这个问题尤为突出。其中一个研究领域涉及法律和法案草案，这些文件可能长达数百页，因此不适合只能处理512个令牌等有限长度文本的模型进行处理。在这篇文章中，我们展示了使用XLM-RoBERTa、Longformer、GPT-3.5及GPT-4模型在比较议程项目（Comparative Agendas Project）多类分类任务中的实验结果，该任务涉及从教育到医疗保健等21个政策主题标签的代码簿。结果显示，专门针对处理长输入文本而预训练的Longformer模型并无显著优势。将GPT变体与表现最佳的开源模型进行比较后发现，后者具有明显的优势。对类别层面因素的分析表明，在处理长篇文本时，特定类别之间的支持和实质重叠对于性能至关重要。
URL
https://arxiv.org/abs/2509.10199
