Abstract
The awareness of multi-cultural human values is critical to the ability of language models (LMs) to generate safe and personalized responses. However, this awareness of LMs has been insufficiently studied, since the computer science community lacks access to the large-scale real-world data about multi-cultural values. In this paper, we present WorldValuesBench, a globally diverse, large-scale benchmark dataset for the multi-cultural value prediction task, which requires a model to generate a rating response to a value question based on demographic contexts. Our dataset is derived from an influential social science project, World Values Survey (WVS), that has collected answers to hundreds of value questions (e.g., social, economic, ethical) from 94,728 participants worldwide. We have constructed more than 20 million examples of the type "(demographic attributes, value question) $\rightarrow$ answer" from the WVS responses. We perform a case study using our dataset and show that the task is challenging for strong open and closed-source models. On merely $11.1\%$, $25.0\%$, $72.2\%$, and $75.0\%$ of the questions, Alpaca-7B, Vicuna-7B-v1.5, Mixtral-8x7B-Instruct-v0.1, and GPT-3.5 Turbo can respectively achieve $<0.2$ Wasserstein 1-distance from the human normalized answer distributions. WorldValuesBench opens up new research avenues in studying limitations and opportunities in multi-cultural value awareness of LMs.
Abstract (translated)
意识到多元文化人类价值观对于语言模型（LMs）生成安全和个性化的回应至关重要。然而，对于LMs的多元文化价值观的意识尚缺乏充分的研究，因为计算机科学领域无法访问关于多元文化价值观的大规模现实世界数据。在本文中，我们提出了一个全球多样、大规模的多文化价值预测任务基准数据集WorldValuesBench，该数据集要求基于人口背景生成一个评分回答，基于 demographic contexts。我们的数据来源于一个著名的社会科学项目World Values Survey（WVS），它收集了来自全球94,728个参与者的数百个价值问题的答案（例如社会、经济、伦理）。我们基于WVS的响应构建了超过2000万对类型"人口属性，价值问题$\rightarrow$答案"的示例。我们使用我们的数据集进行案例研究，并证明了对于强开放和封闭源模型来说，这项任务具有挑战性。仅在11.1%、25.0%、72.2%和75.0%的问题上，Alpaca-7B、Vicuna-7B-v1.5、Mixtral-8x7B-Instruct-v0.1和GPT-3.5 Turbo可以分别实现与人类归一化答案分布的<0.2 Wasserstein 1-距离。WorldValuesBench在研究多元文化价值意识LMs的局限性和机会方面打开了新的研究途径。
URL
https://arxiv.org/abs/2404.16308
