Abstract
Evaluating creative text generation remains a challenge because existing reference-based metrics fail to capture the subjective nature of creativity. We propose a structured evaluation framework for AI story generation comprising four components (Novelty, Value, Adherence, and Resonance) and eleven sub-components. Using controlled story generation via ``Spike Prompting'' and a crowdsourced study of 115 readers, we examine how different creative components shape both immediate and reflective human creativity judgments. Our findings show that creativity is evaluated hierarchically rather than cumulatively, with different dimensions becoming salient at different stages of judgment, and that reflective evaluation substantially alters both ratings and inter-rater agreement. Together, these results support the effectiveness of our framework in revealing dimensions of creativity that are obscured by reference-based evaluation.
Abstract (translated)
评估创意文本生成仍然是一个挑战，因为现有的基于参考的度量标准无法捕捉到创造力的主观性质。我们提出了一种用于AI故事生成的结构化评估框架，该框架由四个组成部分（新颖性、价值、一致性以及共鸣）和十一项子组成部分构成。通过使用“脉冲提示”进行控制的故事生成，并结合115名读者的众包研究，我们探讨了不同的创意组件如何影响人类在即时与反思阶段的创造力评价。我们的发现表明，创造力是分层次而非累加地被评估的，在判断的不同阶段会出现不同维度的重要性。此外，反思性评价显著改变了评分及评判者之间的共识。这些结果共同支持了该框架的有效性，因为它揭示了基于参考评估所忽略的创意维度。
URL
https://arxiv.org/abs/2601.03698
