Abstract
Suicidal ideation detection is critical for real-time suicide prevention, yet its progress faces two under-explored challenges: limited language coverage and unreliable annotation practices. Most available datasets are in English, but even among these, high-quality, human-annotated data remains scarce. As a result, many studies rely on available pre-labeled datasets without examining their annotation process or label reliability. The lack of datasets in other languages further limits the global realization of suicide prevention via artificial intelligence (AI). In this study, we address one of these gaps by constructing a novel Turkish suicidal ideation corpus derived from social media posts and introducing a resource-efficient annotation framework involving three human annotators and two large language models (LLMs). We then address the remaining gaps by performing a bidirectional evaluation of label reliability and model consistency across this dataset and three popular English suicidal ideation detection datasets, using transfer learning through eight pre-trained sentiment and emotion classifiers. These transformers help assess annotation consistency and benchmark model performance against manually labeled data. Our findings underscore the need for more rigorous, language-inclusive approaches to annotation and evaluation in mental health natural language processing (NLP) while demonstrating the questionable performance of popular models with zero-shot transfer learning. We advocate for transparency in model training and dataset construction in mental health NLP, prioritizing data and model reliability.
Abstract (translated)
自杀念头的检测对于实时预防自杀至关重要，然而其进展面临着两个被忽视的挑战：语言覆盖范围有限以及标注实践不可靠。现有的可用数据集大多数是英文，即使在这些数据集中，高质量的人工标注数据仍然稀缺。因此，许多研究依赖于已有的预标注数据集，并未对其标注过程或标签可靠性进行审查。缺乏其他语言的数据集进一步限制了通过人工智能（AI）实现全球自杀预防的进展。 在这项研究中，我们填补了一个空白，构建了一个新颖的土耳其语自杀念头语料库，该语料库来源于社交媒体帖子，并引入了一种资源高效的标注框架，涉及三位人工标注员和两个大型语言模型（LLM）。然后，通过使用八种预训练的情感和情绪分类器进行迁移学习，在此数据集以及三个流行的英语自杀念头检测数据集中执行双向标签可靠性和模型一致性的评估。这些转换器帮助评估注释的一致性，并将模型性能与人工标记的数据相比较。 我们的研究结果强调了在精神健康自然语言处理（NLP）中需要采取更为严谨、多语种包容的方法来进行标注和评估，同时揭示了零样本迁移学习方法中的流行模型存在令人质疑的表现。我们倡导在精神健康NLP领域提高模型训练和数据集构建的透明度，并优先考虑数据和模型的可靠性。
URL
https://arxiv.org/abs/2507.14693
