Abstract
Though patients are increasingly granted digital access to their electronic health records (EHRs), existing interfaces may not support precise, trustworthy answers to patient-specific questions. Large language models (LLM) show promise in clinical question answering (QA), but retrieval-based approaches are computationally inefficient, prone to hallucination, and difficult to deploy over real-life EHRs. In this work, we introduce FHIRPath-QA, the first open dataset and benchmark for patient-specific QA that includes open-standard FHIRPath queries over real-world clinical data. We propose a text-to-FHIRPath QA paradigm that shifts reasoning from free-text generation to FHIRPath query synthesis, significantly reducing LLM usage. Built on MIMIC-IV on FHIR Demo, the dataset pairs over 14k natural language questions in patient and clinician phrasing with validated FHIRPath queries and answers. Further, we demonstrate that state-of-the-art LLMs struggle to deal with ambiguity in patient language and perform poorly in FHIRPath query synthesis. However, they benefit strongly from supervised fine-tuning. Our results highlight that text-to-FHIRPath synthesis has the potential to serve as a practical foundation for safe, efficient, and interoperable consumer health applications, and our dataset and benchmark serve as a starting point for future research on the topic. The full dataset and generation code is available at: this https URL.
Abstract (translated)
尽管患者越来越多地通过数字方式访问其电子健康记录（EHR），但现有的界面可能无法支持对特定患者的精准、可靠的问题回答。大型语言模型（LLM）在临床问答（QA）方面显示出潜力，但由于检索式方法计算效率低、容易产生错误信息并且难以部署到现实生活中的EHR上，因此存在局限性。在这项工作中，我们介绍了FHIRPath-QA，这是第一个公开的数据集和基准测试工具，用于特定患者的问答，并包括在真实世界临床数据上的开放标准FHIRPath查询。我们提出了一种文本到FHIRPath QA的范式，这种范式将推理从自由文本生成转移到了FHIRPath查询合成上，从而大大减少了LLM的使用量。该数据集基于MIMIC-IV on FHIR Demo构建，包含超过14,000个自然语言问题（以患者和临床医生的语言表述），并附有验证过的FHIRPath查询与答案配对。此外，我们还展示了最先进的LLMs在处理患者的模糊表达方面存在困难，并且在生成FHIRPath查询时表现不佳。然而，在监督微调下，它们能够显著受益。 我们的研究结果表明，文本到FHIRPath的合成有可能成为安全、高效和互操作性的消费者健康应用的实际基础，并且我们提供的数据集和基准测试为未来在此领域进行的研究提供了起点。完整的数据集和生成代码可在以下链接获取：[此处插入URL]。
URL
https://arxiv.org/abs/2602.23479
