Abstract
Timing of clinical events is central to characterization of patient trajectories, enabling analyses such as process tracing, forecasting, and causal reasoning. However, structured electronic health records capture few data elements critical to these tasks, while clinical reports lack temporal localization of events in structured form. We present a system that transforms case reports into textual time series-structured pairs of textual events and timestamps. We contrast manual and large language model (LLM) annotations (n=320 and n=390 respectively) of ten randomly-sampled PubMed open-access (PMOA) case reports (N=152,974) and assess inter-LLM agreement (n=3,103; N=93). We find that the LLM models have moderate event recall(O1-preview: 0.80) but high temporal concordance among identified events (O1-preview: 0.95). By establishing the task, annotation, and assessment systems, and by demonstrating high concordance, this work may serve as a benchmark for leveraging the PMOA corpus for temporal analytics.
Abstract (translated)
临床事件的时间定位对于患者轨迹的描述至关重要，它支持过程追踪、预测分析和因果推理等研究。然而，结构化的电子健康记录很少包含这些任务所需的关键数据元素，而临床报告中缺乏以结构化形式表示的事件时间信息。我们提出了一种系统，该系统将病例报告转化为文本时间序列的形式，即将文本中的事件与其发生的时间戳对齐。 我们在10个随机选取的PubMed开放访问（PMOA）案例报告（共包含152,974份报告）上进行了手动标注和大型语言模型（LLM）标注之间的对比研究。具体而言，我们分别使用了320组和390组数据进行手动和LLM标注，并对两个LLM系统之间识别事件的一致性进行了评估（样本量为3,103；总报告数为93）。结果显示，LLM模型在事件召回率方面表现一般(O1-preview: 0.80)，但在所识别事件的时间一致性上表现出色 (O1-preview: 0.95)。 通过确立任务、标注及评估系统，并展示出高度的一致性，这项工作可以作为利用PMOA语料库进行时间序列分析的基准参考。
URL
https://arxiv.org/abs/2504.12350
