Abstract
We consider the problem of distinguishing human-written creative fiction (excerpts from novels) from similar text generated by an LLM. Our results show that, while human observers perform poorly (near chance levels) on this binary classification task, a variety of machine-learning models achieve accuracy in the range 0.93 - 0.98 over a previously unseen test set, even using only short samples and single-token (unigram) features. We therefore employ an inherently interpretable (linear) classifier (with a test accuracy of 0.98), in order to elucidate the underlying reasons for this high accuracy. In our analysis, we identify specific unigram features indicative of LLM-generated text, one of the most important being that the LLM tends to use a larger variety of synonyms, thereby skewing the probability distributions in a manner that is easy to detect for a machine learning classifier, yet very difficult for a human observer. Four additional explanation categories were also identified, namely, temporal drift, Americanisms, foreign language usage, and colloquialisms. As identification of the AI-generated text depends on a constellation of such features, the classification appears robust, and therefore not easy to circumvent by malicious actors intent on misrepresenting AI-generated text as human work.
Abstract (translated)
我们研究了区分人类创作的创意小说（小说片段）和类似由大型语言模型生成文本的问题。我们的结果显示，虽然人类观察者在这项二元分类任务中表现不佳（接近随机水平），但多种机器学习模型在对先前未见过的数据集进行测试时达到了0.93到0.98之间的准确率，即使只使用短样本和单词特征（一元特征）也是如此。因此，我们采用了一种固有的可解释的线性分类器（其测试精度为0.98），以揭示这种高准确性背后的原因。在我们的分析中，我们确定了一些特定的一元特征，这些特征表明了由大型语言模型生成的文本的特点，其中最重要的是：大型语言模型倾向于使用更多种类的同义词，从而使概率分布向易于机器学习分类器检测但对人类观察者来说非常难以察觉的方向偏移。此外还确定了四个额外的解释类别，即时间漂移、美式英语表达、外语用法和俚语。由于识别AI生成文本依赖于多种此类特征的组合，因此这种分类显得十分稳健，并且不容易被试图将AI生成内容冒充为人类作品的恶意行为者所规避。
URL
https://arxiv.org/abs/2601.07368
