Abstract
Large language models (LLMs) are increasingly tasked with producing and manipulating structured artifacts. We consider the task of end-to-end spreadsheet generation, where language models are prompted to produce spreadsheet artifacts to satisfy users' explicit and implicit constraints, specified in natural language. We introduce SpreadsheetArena, a platform for evaluating models' performance on the task via blind pairwise evaluations of LLM-generated spreadsheet workbooks. As with other complex, open-ended tasks, relevant evaluation criteria can vary substantially across use cases and prompts, often in ways that are difficult to formalize. Compared to general chat or text generation settings, spreadsheet generation presents unique challenges and opportunities: the task output structure is well-defined and multi-dimensional, and there are often complex considerations around interactivity and layout. Among other findings, we observe that stylistic, structural, and functional features of preferred spreadsheets vary substantially across use cases, and expert evaluations of spreadsheets for finance prompts suggests that even highly ranked arena models do not reliably produce spreadsheets aligned with domain-specific best practices. Our hope is that our work prompts further study of end-to-end spreadsheet generation as a challenging and interesting category of complex, open-ended tasks for LLMs. Our live arena is hosted at this https URL.
Abstract (translated)
大型语言模型（LLMs）越来越多地被用于生成和处理结构化数据。我们研究了一个端到端的电子表格生成任务，在这个任务中，语言模型根据自然语言提示来生成满足用户明确或隐含需求的电子表格。为此，我们引入了SpreadsheetArena平台，这是一个通过盲对评估（blind pairwise evaluations）来评价LLM生成电子表格工作簿表现的平台。 与其他复杂的开放式任务一样，相关的评判标准可能会因使用案例和提示的不同而有很大差异，且这些标准往往难以形式化。与一般的聊天或文本生成设置相比，电子表格生成具有独特的挑战性和机遇：其输出结构是明确且多维度的，并且常常存在复杂的人机交互以及布局方面的考量。 我们的研究发现，不同应用场景中用户偏好电子表格的风格、结构和功能特征差异显著；而对于财务相关提示的专家评估则表明，即使是排名靠前的模型生成的电子表格也未必符合特定领域的最佳实践。我们希望这项工作能够激发更多关于端到端电子表格生成作为复杂开放式任务领域中的挑战性话题的研究。 我们的在线竞赛平台可在此 [链接]（请将"this https URL"替换为实际的有效URL）访问。
URL
https://arxiv.org/abs/2603.10002
