Abstract
Storytelling is a fundamental aspect of human communication, relying heavily on creativity to produce narratives that are novel, appropriate, and surprising. While large language models (LLMs) have recently demonstrated the ability to generate high-quality stories, their creative capabilities remain underexplored. Previous research has either focused on creativity tests requiring short responses or primarily compared model performance in story generation to that of professional writers. However, the question of whether LLMs exhibit creativity in writing short stories on par with the average human remains unanswered. In this work, we conduct a systematic analysis of creativity in short story generation across LLMs and everyday people. Using a five-sentence creative story task, commonly employed in psychology to assess human creativity, we automatically evaluate model- and human-generated stories across several dimensions of creativity, including novelty, surprise, and diversity. Our findings reveal that while LLMs can generate stylistically complex stories, they tend to fall short in terms of creativity when compared to average human writers.
Abstract (translated)
讲故事是人类交流的基本方面，它很大程度上依赖于创造力来生成新颖、适当且令人惊讶的故事。虽然大型语言模型（LLMs）最近展示了生成高质量故事的能力，但它们的创造性能力仍处于未充分探索的状态。先前的研究要么集中在需要简短回答的创意测试上，要么主要比较了模型在故事生成方面的表现与专业作家的表现。然而，关于LLMs是否能在创作短篇小说方面表现出与普通人相当的创造力这一问题仍未得到解答。在这项工作中，我们对跨LLMs和普通人的短篇小说生成中的创造性进行了系统的分析。使用心理学中常用的五个句子创意故事任务来自动评估模型和人类生成的故事在多个创造性维度上的表现，包括新颖性、惊喜度和多样性。我们的发现表明，尽管LLMs能够生成风格复杂的故事情节，但与普通人作家相比，在创造力方面仍有所不足。
URL
https://arxiv.org/abs/2411.02316
