Abstract
We present a comprehensive evaluation of gender fairness in large language models (LLMs), focusing on their ability to handle both binary and non-binary genders. While previous studies primarily focus on binary gender distinctions, we introduce the Gender Inclusivity Fairness Index (GIFI), a novel and comprehensive metric that quantifies the diverse gender inclusivity of LLMs. GIFI consists of a wide range of evaluations at different levels, from simply probing the model with respect to provided gender pronouns to testing various aspects of model generation and cognitive behaviors under different gender assumptions, revealing biases associated with varying gender identifiers. We conduct extensive evaluations with GIFI on 22 prominent open-source and proprietary LLMs of varying sizes and capabilities, discovering significant variations in LLMs' gender inclusivity. Our study highlights the importance of improving LLMs' inclusivity, providing a critical benchmark for future advancements in gender fairness in generative models.
Abstract (translated)
我们提出了一种关于大型语言模型（LLM）性别公平性的全面评估，重点考察它们处理二元和非二元性别的能力。尽管以往的研究主要集中在二元性别区分上，我们引入了性别包容性公平指数（GIFI），这是一个新颖且全面的度量标准，用于量化LLM在不同性别标识下的多样性与包容性。GIFI涵盖了从简单地用提供的性别代词来探测模型到测试模型在不同性别假设下生成和认知行为的各种方面的广泛评估，揭示了与各种性别标识相关的偏见。我们在22个不同的开源和专有LLM上进行了广泛的GIFI评估，这些模型的大小和能力各异，发现它们在性别包容性方面存在显著差异。我们的研究强调了提高LLM包容性的必要性，并为未来生成模型中性别公平性的进展提供了重要的基准。
URL
https://arxiv.org/abs/2506.15568
