Abstract
Large Language Models (LLMs) are increasingly utilized for mental health support; however, current safety benchmarks often fail to detect the complex, longitudinal risks inherent in therapeutic dialogue. We introduce an evaluation framework that pairs AI psychotherapists with simulated patient agents equipped with dynamic cognitive-affective models and assesses therapy session simulations against a comprehensive quality of care and risk ontology. We apply this framework to a high-impact test case, Alcohol Use Disorder, evaluating six AI agents (including ChatGPT, Gemini, and this http URL) against a clinically-validated cohort of 15 patient personas representing diverse clinical phenotypes. Our large-scale simulation (N=369 sessions) reveals critical safety gaps in the use of AI for mental health support. We identify specific iatrogenic risks, including the validation of patient delusions ("AI Psychosis") and failure to de-escalate suicide risk. Finally, we validate an interactive data visualization dashboard with diverse stakeholders, including AI engineers and red teamers, mental health professionals, and policy experts (N=9), demonstrating that this framework effectively enables stakeholders to audit the "black box" of AI psychotherapy. These findings underscore the critical safety risks of AI-provided mental health support and the necessity of simulation-based clinical red teaming before deployment.
Abstract (translated)
大型语言模型（LLM）在心理健康支持中的应用越来越广泛；然而，现有的安全基准通常无法检测出治疗对话中内在的复杂和长期风险。我们引入了一个评估框架，该框架将AI心理治疗师与装备有动态认知-情感模型的模拟患者代理配对，并根据全面的质量护理和风险本体论来评估治疗会话模拟的效果。我们将此框架应用于一个具有重大影响的测试案例——酒精使用障碍（Alcohol Use Disorder），评估六个AI代理（包括ChatGPT、Gemini以及未指明的另一个网址）在15个代表多样化临床表现的人格模型（clinically-validated cohort of 15 patient personas representing diverse clinical phenotypes）上的表现。我们进行了大规模模拟实验（N=369次会话），揭示了在心理健康支持中使用AI的关键安全漏洞。我们识别出特定的医源性风险，包括验证患者妄想症（"AI精神病"）和未能降低自杀风险。 最终，我们与各种利益相关者合作，包括AI工程师、红队成员、心理健康专业人士以及政策专家（N=9），验证了一个互动数据可视化仪表板的有效性。这表明此框架能够使利益相关方审计“黑箱”中的AI心理治疗过程。这些发现强调了由AI提供的心理健康支持的关键安全风险，并且在部署之前进行基于模拟的临床红队测试是必要的。
URL
https://arxiv.org/abs/2602.19948
