Abstract
As Large Language Models (LLMs) are increasingly popularized in the multilingual world, ensuring hallucination-free factuality becomes markedly crucial. However, existing benchmarks for evaluating the reliability of Multimodal Large Language Models (MLLMs) predominantly focus on textual or visual modalities with a primary emphasis on English, which creates a gap in evaluation when processing multilingual input, especially in speech. To bridge this gap, we propose a novel \textbf{C}ross-lingual and \textbf{C}ross-modal \textbf{F}actuality benchmark (\textbf{CCFQA}). Specifically, the CCFQA benchmark contains parallel speech-text factual questions across 8 languages, designed to systematically evaluate MLLMs' cross-lingual and cross-modal factuality capabilities. Our experimental results demonstrate that current MLLMs still face substantial challenges on the CCFQA benchmark. Furthermore, we propose a few-shot transfer learning strategy that effectively transfers the Question Answering (QA) capabilities of LLMs in English to multilingual Spoken Question Answering (SQA) tasks, achieving competitive performance with GPT-4o-mini-Audio using just 5-shot training. We release CCFQA as a foundational research resource to promote the development of MLLMs with more robust and reliable speech understanding capabilities. Our code and dataset are available at this https URL.
Abstract (translated)
随着大型语言模型（LLM）在多语言世界中越来越受欢迎，确保无幻觉的事实性变得尤为重要。然而，目前评估多模态大型语言模型（MLLMs）可靠性的现有基准主要集中在文本或视觉模式上，并且以英语为主，这在处理多种语言的输入时，尤其是语音输入时，造成了评估上的缺口。为弥补这一差距，我们提出了一种新颖的跨语言和跨模态事实性基准测试（CCFQA）。具体来说，CCFQA 基准包含了跨越8种语言的平行语音-文本事实问题，旨在系统地评估MLLMs在跨语言和跨模态方面的事实能力。我们的实验结果表明，当前的MLLMs 在CCFQA基准测试中仍面临重大挑战。此外，我们还提出了一种基于少量样本的学习策略，可以有效地将英语环境中大型语言模型（LLM）的问题回答（QA）能力转移到多语言口语问答（SQA）任务上，并且仅通过5次训练就能达到与GPT-4o-mini-Audio相当的性能水平。我们将CCFQA作为基础研究资源发布，以促进具有更强大和可靠语音理解能力的MLLMs的发展。我们的代码和数据集可在[此链接](https://this-url.com/)获取。
URL
https://arxiv.org/abs/2508.07295
