Abstract
This study compares the performance of two open-source large language models (LLMs)-Llama3-70B and DeepSeekR1-distill-Llama3-70B-on six biomedical text classification tasks. Four tasks involve data from social media, while two tasks focus on clinical notes from electronic health records, and all experiments were performed in zero-shot settings. Performance metrics, including precision, recall, and F1 scores, were measured for each task, along with their 95% confidence intervals. Results demonstrated that DeepSeekR1-distill-Llama3-70B generally performs better in terms of precision on most tasks, with mixed results on recall. While the zero-shot LLMs demonstrated high F1 scores for some tasks, they grossly underperformed on others, for data from both sources. The findings suggest that model selection should be guided by the specific requirements of the health-related text classification tasks, particularly when considering the precision-recall trade-offs, and that, in the presence of annotated data, supervised classification approaches may be more reliable than zero-shot LLMs.
Abstract (translated)
这项研究比较了两个开源大型语言模型（LLMs）——Llama3-70B 和 DeepSeekR1-distill-Llama3-70B 在六个生物医学文本分类任务中的性能。其中四个任务涉及社交媒体数据，而另外两个任务则侧重于电子健康记录中的临床笔记，并且所有实验均在零样本设置下进行。每个任务的性能指标包括准确率、召回率和F1值，以及95%置信区间。 结果显示，在大多数任务中，DeepSeekR1-distill-Llama3-70B 在准确率方面表现更好，而对召回率的结果则较为混合。尽管零样本LLMs在某些任务上表现出较高的F1分数，但在另一些任务上的表现明显不足，无论是社交媒体数据还是电子健康记录中的数据。 研究结果表明，在选择模型时应考虑特定的健康相关文本分类任务的具体需求，特别是在权衡准确率和召回率方面。此外，当存在标注数据时，监督分类方法可能比零样本LLMs更为可靠。
URL
https://arxiv.org/abs/2503.15169
