Abstract
For a long time, the authorship of the Federalist Papers had been a subject of inquiry and debate, not only by linguists and historians but also by statisticians. In what was arguably the first Bayesian case study, Mosteller and Wallace (1963) provided the first statistical evidence for attributing all disputed papers to Madison. Our paper revisits this historical dataset but from a lens of modern language models, both small and large. We review some of the more popular Large Language Model (LLM) tools and examine them from a statistical point of view in the context of text classification. We investigate whether, without any attempt to fine-tune, the general embedding constructs can be useful for stylometry and attribution. We explain differences between various word/phrase embeddings and discuss how to aggregate them in a document. Contrary to our expectations, we exemplify that dimension expansion with word embeddings may not always be beneficial for attribution relative to dimension reduction with topic embeddings. Our experiments demonstrate that default LLM embeddings (even after manual fine-tuning) may not consistently improve authorship attribution accuracy. Instead, Bayesian analysis with topic embeddings trained on ``function words" yields superior out-of-sample classification performance. This suggests that traditional (small) statistical language models, with their interpretability and solid theoretical foundation, can offer significant advantages in authorship attribution tasks. The code used in this analysis is available at this http URL
Abstract (translated)
长期以来，《联邦党人论文》的作者身份一直是语言学家、历史学家乃至统计学家的研究和讨论对象。Mosteller 和 Wallace (1963) 的研究可以说是第一个贝叶斯案例研究，他们提供了将所有有争议的文章归于麦迪逊的第一份统计证据。本文重新审视了这个历史性数据集，但采用的是现代语言模型的视角，无论是小型还是大型模型。 我们回顾了一些较为流行的大型语言模型（LLM）工具，并从统计学的角度在文本分类的背景下对其进行研究。我们探讨了未经任何微调的情况下，通用嵌入结构是否可以用于风格计量和归因。我们将解释不同词/短语嵌入之间的差异，并讨论如何在文档中聚合它们。与我们的预期相反，我们举例说明了使用词嵌入进行维度扩展可能并不总是比使用主题嵌入进行维度缩减对作者归因更有益。 实验表明，默认的大型语言模型嵌入（即使经过手动微调）也不能一贯提高作者身份识别准确性。相比之下，基于“功能词”训练的主题嵌入与贝叶斯分析相结合，在外样本分类性能上表现出色。这表明传统的（小型）统计语言模型，凭借其可解释性和坚实的理论基础，可以在作者归因任务中提供显著优势。 本研究使用的代码可在以下链接获取：[此 HTTP URL]
URL
https://arxiv.org/abs/2503.01869
