Abstract
Current methods for evaluating large language models (LLMs) typically focus on high-level tasks such as text generation, without targeting a particular AI application. This approach is not sufficient for evaluating LLMs for Responsible AI dimensions like fairness, since protected attributes that are highly relevant in one application may be less relevant in another. In this work, we construct a dataset that is driven by a real-world application (generate a plain-text product description, given a list of product features), parameterized by fairness attributes intersected with gendered adjectives and product categories, yielding a rich set of labeled prompts. We show how to use the data to identify quality, veracity, safety, and fairness gaps in LLMs, contributing a proposal for LLM evaluation paired with a concrete resource for the research community.
Abstract (translated)
目前评估大型语言模型（LLMs）的方法通常集中在如文本生成等高层次任务上，而没有针对特定的人工智能应用程序。这种方法不足以对像公平性这样的负责任AI维度进行评价，因为在某一应用中非常相关的受保护属性，在其他应用中可能不太相关。在本项工作中，我们构建了一个由真实世界应用场景驱动的数据集（根据产品特征列表生成纯文本的产品描述），该数据集通过与性别形容词和产品类别相交的公平性属性参数化，产生了一系列丰富的带有标签的提示。我们展示了如何使用这些数据来识别LLMs的质量、真实性、安全性和公平性的差距，并为研究社区提出了一个结合具体资源的评估方法提案。
URL
https://arxiv.org/abs/2510.20782
