Abstract
Large Language Models (LLMs) have demonstrated potential in predicting mental health outcomes from online text, yet traditional classification methods often lack interpretability and robustness. This study evaluates structured reasoning techniques-Chain-of-Thought (CoT), Self-Consistency (SC-CoT), and Tree-of-Thought (ToT)-to improve classification accuracy across multiple mental health datasets sourced from Reddit. We analyze reasoning-driven prompting strategies, including Zero-shot CoT and Few-shot CoT, using key performance metrics such as Balanced Accuracy, F1 score, and Sensitivity/Specificity. Our findings indicate that reasoning-enhanced techniques improve classification performance over direct prediction, particularly in complex cases. Compared to baselines such as Zero Shot non-CoT Prompting, and fine-tuned pre-trained transformers such as BERT and Mental-RoBerta, and fine-tuned Open Source LLMs such as Mental Alpaca and Mental-Flan-T5, reasoning-driven LLMs yield notable gains on datasets like Dreaddit (+0.52\% over M-LLM, +0.82\% over BERT) and SDCNL (+4.67\% over M-LLM, +2.17\% over BERT). However, performance declines in Depression Severity, and CSSRS predictions suggest dataset-specific limitations, likely due to our using a more extensive test set. Among prompting strategies, Few-shot CoT consistently outperforms others, reinforcing the effectiveness of reasoning-driven LLMs. Nonetheless, dataset variability highlights challenges in model reliability and interpretability. This study provides a comprehensive benchmark of reasoning-based LLM techniques for mental health text classification. It offers insights into their potential for scalable clinical applications while identifying key challenges for future improvements.
Abstract (translated)
大型语言模型（LLMs）在从在线文本预测心理健康结果方面展示了潜力，但传统的分类方法往往缺乏可解释性和鲁棒性。本研究评估了结构化推理技术——链式思维（CoT）、自我一致性（SC-CoT）和思想树（ToT），以提高来自Reddit的多个心理健康数据集上的分类准确性。我们分析了包括零样本链式思维和少量样本链式思维在内的基于推理的提示策略，并使用平衡准确率、F1分数以及敏感度/特异度等关键性能指标进行评估。我们的研究发现表明，增强推理的技术比直接预测在复杂案例中提高了分类性能。与基线模型（如零样本非CoT提示和预训练转换器BERT及Mental-RoBerta）相比，以及与开源LLMs（如Mental Alpaca和Mental-Flan-T5）微调后的结果相比，基于推理的LLMs在Dreaddit数据集上提高了0.52%（相对于M-LLM）和0.82%（相对于BERT），在SDCNL数据集上分别提升了4.67%（相对于M-LLM）和2.17%（相对于BERT）。然而，在抑郁症严重程度以及CSSRS预测中的性能下降表明了特定于数据集的限制，这可能是由于使用了更广泛的数据测试集。在提示策略中，少量样本链式思维一贯优于其他方法，进一步验证了基于推理的LLMs的有效性。尽管如此，数据集的变化突显了模型可靠性和可解释性的挑战。 本研究为基于推理的LLM技术在心理健康文本分类上的全面基准提供了依据，并为其在未来临床应用中的潜力和面临的关键挑战提供了见解。
URL
https://arxiv.org/abs/2503.10095
