Abstract
Autonomous driving requires reliable reasoning over fine-grained 3D scene facts. Fine-grained question answering over multi-modal driving observations provides a natural way to evaluate this capability, yet existing perception pipelines and driving-oriented large language model (LLM) methods still suffer from unreliable scene facts, hallucinations, opaque reasoning, and heavy reliance on task-specific training. We present KLDrive, the first knowledge-graph-augmented LLM reasoning framework for fine-grained question answering in autonomous driving. KLDrive addresses this problem through designing two tightly coupled components: an energy-based scene fact construction module that consolidates multi-source evidence into a reliable scene knowledge graph, and an LLM agent that performs fact-grounded reasoning over a constrained action space under explicit structural constraints. By combining structured prompting with few-shot in-context exemplars, the framework adapts to diverse reasoning tasks without heavy task-specific fine-tuning. Experiments on two large-scale autonomous-driving QA benchmarks show that KLDrive outperforms prior state-of-the-art methods, achieving the best overall accuracy of 65.04% on NuScenes-QA and the best SPICE score of 42.45 on GVQA. On counting, the most challenging factual reasoning task, it improves over the strongest baseline by 46.01 percentage points, demonstrating substantially reduced hallucinations and the benefit of coupling reliable scene fact construction with explicit reasoning.
Abstract (translated)
自动驾驶需要对精细的3D场景事实进行可靠推理。基于多模态驾驶观测的细粒度问答为评估这一能力提供了自然途径，但现有感知流程和面向驾驶的大语言模型（LLM）方法仍受困于不可靠的场景事实、幻觉、不透明的推理过程以及对任务特定训练的过度依赖。我们提出KLDrive——首个用于自动驾驶细粒度问答的知识图谱增强型LLM推理框架。该框架通过设计两个紧密耦合的组件来解决这一问题：一是基于能量的场景事实验证模块，可将多源证据整合为可靠的场景知识图谱；二是LLM智能体，能在显式结构约束下于受限动作空间中进行基于事实的推理。通过结合结构化提示与少量上下文示例，该框架无需大量任务特定微调即可适应多样化的推理任务。在两个大规模自动驾驶问答基准测试上的实验表明，KLDrive超越了先前最先进方法，在NuScenes-QA上取得65.04%的最佳总体准确率，在GVQA上取得42.45的最佳SPICE分数。在最具挑战性的计数任务（一种事实推理任务）上，其表现比最强基线提升46.01个百分点，显著降低了幻觉现象，并验证了将可靠场景事实验证与显式推理相结合的优势。
URL
https://arxiv.org/abs/2603.21029
