Abstract
Mental health remains a challenging problem all over the world, with issues like depression, anxiety becoming increasingly common. Large Language Models (LLMs) have seen a vast application in healthcare, specifically in answering medical questions. However, there is a lack of standard benchmarking datasets for question answering (QA) in mental health. Our work presents a novel multiple choice dataset, MHQA (Mental Health Question Answering), for benchmarking Language models (LMs). Previous mental health datasets have focused primarily on text classification into specific labels or disorders. MHQA, on the other hand, presents question-answering for mental health focused on four key domains: anxiety, depression, trauma, and obsessive/compulsive issues, with diverse question types, namely, factoid, diagnostic, prognostic, and preventive. We use PubMed abstracts as the primary source for QA. We develop a rigorous pipeline for LLM-based identification of information from abstracts based on various selection criteria and converting it into QA pairs. Further, valid QA pairs are extracted based on post-hoc validation criteria. Overall, our MHQA dataset consists of 2,475 expert-verified gold standard instances called MHQA-gold and ~56.1k pairs pseudo labeled using external medical references. We report F1 scores on different LLMs along with few-shot and supervised fine-tuning experiments, further discussing the insights for the scores.
Abstract (translated)
心理健康仍然是全球面临的严峻问题，抑郁症和焦虑症等状况变得越来越普遍。大型语言模型（LLMs）在医疗领域得到了广泛的应用，尤其是在回答医学相关的问题方面。然而，在心理健康领域的问答基准数据集存在不足。我们的研究提出了一套新型的多选题数据集MHQA（心理健康问答），用于评估语言模型在这一特定领域的能力。 以往的心理健康数据集主要集中在文本分类到特定标签或疾病上，而MHQA则专注于四个关键领域的心理健康问题解答：焦虑、抑郁、创伤和强迫/侵扰障碍，并涵盖多种问题类型，包括事实性、诊断性、预后性和预防性。我们使用PubMed摘要作为问答的主要信息来源。 为了创建这一数据集，我们开发了一个严格的流程，利用大型语言模型从摘要中识别出符合各种选择标准的信息，并将其转化为问答对。此外，通过后期验证准则来提取有效的问题答案对。总的来说，我们的MHQA数据集包含2,475个经过专家验证的金标准实例（即MHQA-gold）和约56,100对使用外部医学参考进行伪标记的数据。 我们报告了不同语言模型在这一任务上的F1分数，并讨论了少量样本和监督微调实验中的得分见解。
URL
https://arxiv.org/abs/2502.15418
