Abstract
Objective: Zero-shot methodology promises to cut down on costs of dataset annotation and domain expertise needed to make use of NLP. Generative large language models trained to align with human goals have achieved high zero-shot performance across a wide variety of tasks. As of yet, it is unclear how well these models perform on biomedical relation extraction (RE). To address this knowledge gap, we explore patterns in the performance of OpenAI LLMs across a diverse sampling of RE tasks. Methods: We use OpenAI GPT-4-turbo and their reasoning model o1 to conduct end-to-end RE experiments on seven datasets. We use the JSON generation capabilities of GPT models to generate structured output in two ways: (1) by defining an explicit schema describing the structure of relations, and (2) using a setting that infers the structure from the prompt language. Results: Our work is the first to study and compare the performance of the GPT-4 and o1 for the end-to-end zero-shot biomedical RE task across a broad array of datasets. We found the zero-shot performances to be proximal to that of fine-tuned methods. The limitations of this approach are that it performs poorly on instances containing many relations and errs on the boundaries of textual mentions. Conclusion: Recent large language models exhibit promising zero-shot capabilities in complex biomedical RE tasks, offering competitive performance with reduced dataset curation and NLP modeling needs at the cost of increased computing, potentially increasing medical community accessibility. Addressing the limitations we identify could further boost reliability. The code, data, and prompts for all our experiments are publicly available: this https URL
Abstract (translated)
目标：零样本方法承诺减少数据集标注成本和使用自然语言处理（NLP）所需的专业知识。经过训练以与人类目标对齐的生成式大规模语言模型在各种任务中实现了高水平的零样本性能。然而，到目前为止，这些模型在生物医学关系抽取（RE）方面的表现尚不清楚。为了填补这一知识空白，我们探索了OpenAI LLMs在一系列多样化的RE任务中的性能模式。 方法：我们使用OpenAI GPT-4-turbo和推理模型o1对七种数据集进行端到端的零样本生物医学关系抽取实验。我们利用GPT模型的JSON生成能力以两种方式生成结构化输出：（1）通过定义描述关系结构的显式模式；（2）使用从提示语言中推断出结构的设置。 结果：我们的研究是首次对GPT-4和o1在广泛的生物医学RE数据集上进行端到端零样本任务性能的研究与比较。我们发现这些模型的零样本表现接近于经过微调的方法的表现。然而，该方法存在局限性，在包含许多关系的实例中表现较差，并且在文本提及边界的处理上有偏差。 结论：最近的大规模语言模型在复杂的生物医学RE任务中表现出令人鼓舞的零样本能力，减少了数据集整理和NLP建模的需求，但增加了计算成本，这可能提高医疗社区的可访问性。解决我们识别出的限制可能会进一步增强其可靠性。所有实验的代码、数据和提示均公开提供：[此URL](this https URL)
URL
https://arxiv.org/abs/2504.04083
