Abstract
With the rise of large language models (LLMs), they have become instrumental in applications such as Retrieval-Augmented Generation (RAG). Yet evaluating these systems remains bottlenecked by the time and cost of building specialized assessment datasets. We introduce KNIGHT, an LLM-based, knowledge-graph-driven framework for generating multiple-choice question (MCQ) datasets from external sources. KNIGHT constructs a topic-specific knowledge graph, a structured and parsimonious summary of entities and relations, that can be reused to generate instructor-controlled difficulty levels, including multi-hop questions, without repeatedly re-feeding the full source text. This knowledge graph acts as a compressed, reusable state, making question generation a cheap read over the graph. We instantiate KNIGHT on Wikipedia/Wikidata while keeping the framework domain- and ontology-agnostic. As a case study, KNIGHT produces six MCQ datasets in History, Biology, and Mathematics. We evaluate quality on five criteria: fluency, unambiguity (single correct answer), topic relevance, option uniqueness, and answerability given the provided sources (as a proxy for hallucination). Results show that KNIGHT enables token- and cost-efficient generation from a reusable graph representation, achieves high quality across these criteria, and yields model rankings aligned with MMLU-style benchmarks, while supporting topic-specific and difficulty-controlled evaluation.
Abstract (translated)
随着大型语言模型（LLMs）的兴起，它们在诸如检索增强生成（RAG）等应用中变得至关重要。然而，评估这些系统的瓶颈仍然是构建专门评估数据集所需的时间和成本。我们介绍了一种名为KNIGHT的框架，它是一种基于LLM的知识图谱驱动的方法，可以从外部来源生成多项选择题（MCQ）数据集。KNIGHT通过构造一个特定主题的知识图谱来工作，这个图谱是实体及其关系的结构化且精简的摘要，可以被重复用于根据教师控制的难度级别生成问题，包括多跳问题，并不需要反复重新输入完整的源文本。这种知识图谱充当压缩、可重用的状态，使得基于图谱的读取进行问题生成成本低廉。 在维基百科/维基数据上实现KNIGHT时，我们保持了框架对领域和本体论的中立性。作为案例研究，KNIGHT产生了六个历史、生物学和数学领域的MCQ数据集。我们在五个标准下评估其质量：流畅度、无歧义（只有一个正确答案）、主题相关性、选项的独特性和给定来源下的可回答性（作为幻觉的代理）。结果表明，KNIGHT能够从可重用图表示中以成本效益的方式生成高质量的问题，并在这些标准上达到了高水平的质量。同时，它还支持特定主题和难度级别的评估，与MMLU风格基准测试中的模型排名一致。
URL
https://arxiv.org/abs/2602.20135
