Abstract
Querying generative AI models, e.g., large language models (LLMs), has become a prevalent method for information acquisition. However, existing query-answer datasets primarily focus on textual responses, making it challenging to address complex user queries that require visual demonstrations or explanations for better understanding. To bridge this gap, we construct a benchmark, RealVideoQuest, designed to evaluate the abilities of text-to-video (T2V) models in answering real-world, visually grounded queries. It identifies 7.5K real user queries with video response intents from Chatbot-Arena and builds 4.5K high-quality query-video pairs through a multistage video retrieval and refinement process. We further develop a multi-angle evaluation system to assess the quality of generated video answers. Experiments indicate that current T2V models struggle with effectively addressing real user queries, pointing to key challenges and future research opportunities in multimodal AI.
Abstract (translated)
查询生成式人工智能模型（如大型语言模型，LLMs）已成为信息获取的一种常见方法。然而，现有的查询-回答数据集主要侧重于文本响应，这使得难以解决那些需要视觉演示或解释以更好地理解的复杂用户问题成为挑战。为了弥合这一差距，我们构建了一个基准测试框架 RealVideoQuest，旨在评估文字转视频（T2V）模型在回答真实世界、基于视觉的问题时的能力。该框架从 Chatbot-Arena 中识别出具有视频回应意图的 7,500 条真实用户查询，并通过多阶段视频检索和优化过程构建了高质量的查询-视频对共计 4,500 对。此外，我们还开发了一个多角度评估系统来衡量生成的视频回答的质量。实验表明，当前的文字转视频模型在有效处理真实用户的查询方面存在困难，这指出了多模态人工智能领域中的关键挑战和未来研究机会。
URL
https://arxiv.org/abs/2506.01689
