Abstract
Clinicians often need to retrieve patient-specific information from electronic health records (EHRs), a task that is time-consuming and error-prone. We present a locally deployable Clinical Contextual Question Answering (CCQA) framework that answers clinical questions directly from EHRs without external data transfer. Open-source large language models (LLMs) ranging from 4B to 70B parameters were benchmarked under fully offline conditions using 1,664 expert-annotated question-answer pairs derived from records of 183 patients. The dataset consisted predominantly of Finnish clinical text. In free-text generation, Llama-3.1-70B achieved 95.3% accuracy and 97.3% consistency across semantically equivalent question variants, while the smaller Qwen3-30B-A3B-2507 model achieved comparable performance. In a multiple-choice setting, models showed similar accuracy but variable calibration. Low-precision quantization (4-bit and 8-bit) preserved predictive performance while reducing GPU memory requirements and improving deployment feasibility. Clinical evaluation identified clinically significant errors in 2.9% of outputs, and semantically equivalent questions occasionally yielded discordant responses, including instances where one formulation was correct and the other contained a clinically significant error (0.96% of cases). These findings demonstrate that locally hosted open-source LLMs can accurately retrieve patient-specific information from EHRs using natural-language queries, while highlighting the need for validation and human oversight in clinical deployment.
Abstract (translated)
临床医生经常需要从电子健康记录（EHR）中检索患者特定信息，这一任务耗时且容易出错。我们提出了一种可本地部署的临床上下文问答（CCQA）框架，该框架无需外部数据传输即可直接从EHR中回答临床问题。研究在完全离线条件下，使用来自183名患者记录的1664对专家标注问答数据，对参数规模从4B到70B的开源大语言模型（LLM）进行了基准测试。数据集主要由芬兰临床文本构成。在自由文本生成任务中，Llama-3.1-70B在语义等价的问题变体上达到了95.3%的准确率和97.3%的一致性，而较小的Qwen3-30B-A3B-2507模型也实现了 comparable performance。在多项选择题设置中，模型表现出相似的准确率但校准程度各异。低精度量化（4位和8位）在保持预测性能的同时降低了GPU内存需求，提升了部署可行性。临床评估发现2.9%的输出存在临床显著错误，且语义等价问题偶尔会产生矛盾响应——包括一种表述正确而另一种包含临床显著错误的情况（占0.96%）。这些 findings表明，本地托管的开源LLM能够使用自然语言查询从EHR中准确检索患者特定信息，同时也凸显了临床部署中验证和人工监督的必要性。
URL
https://arxiv.org/abs/2603.26434
