Abstract
Large Language Models (LLMs) demonstrate varying performance across languages and cultural contexts. This study introduces a novel, culturally-rich, multilingual dataset derived from video recordings of the Romanian game show "Who Wants to Be a Millionaire?" (Vrei sÄ fii Milionar?). We employed an innovative process combining optical character recognition (OCR), automated text extraction, and manual verification to collect question-answer pairs, enriching them with metadata including question domain (e.g., biology, history), cultural relevance (Romanian-specific vs. international), and difficulty. Benchmarking state-of-the-art LLMs, including Romanian-adapted models, on this dataset revealed significant performance disparities: models consistently achieve higher accuracy (80-95%) on international questions compared to Romanian-specific cultural questions (50-75%). We further investigate these differences through experiments involving machine translation of Romanian questions into English and cross-lingual tests using a comparable dataset in French. Our findings underscore the impact of cultural context and data source on LLM performance and offer practical insights for building robust, culturally-aware multilingual NLP systems, especially in educational domains. The dataset is publicly available at Hugging Face.
Abstract (translated)
大型语言模型（LLMs）在不同语言和文化背景下的表现存在差异。本研究引入了一个新颖且文化底蕴丰富的多语言数据集，该数据集源自罗马尼亚电视益智节目《谁想成为百万富翁？》（Vrei să fii milionar?）的视频记录。我们采用了一种创新的方法结合光学字符识别（OCR）、自动文本提取和人工验证来收集问题-答案对，并且通过添加包括问题领域（例如生物、历史）、文化相关性（特定于罗马尼亚的文化内容与国际通用的内容）以及难度等元数据，进一步丰富了这些数据。 在使用这一数据集评估当前最先进的LLM模型（包括经过罗马尼亚语适应的模型）时，我们发现显著的表现差距：所有模型在处理具有国际普遍性的题目时准确率较高（80-95%），而在面对特定于罗马尼亚文化的题目时则表现较差（50-75%）。为了进一步探讨这一差异，我们进行了实验，将罗马尼亚语问题通过机器翻译成英语，并使用与该数据集相似的法语文本进行跨语言测试。我们的发现强调了文化背景和数据来源对LLM性能的影响，并为构建稳健且具有文化意识的多语言自然语言处理（NLP）系统提供了实际见解，特别是在教育领域。 该数据集已在Hugging Face上公开发布。
URL
https://arxiv.org/abs/2506.05991
