Abstract
Agents represent one of the most emerging applications of Large Language Models (LLMs) and Generative AI, with their effectiveness hinging on multimodal capabilities to navigate complex user environments. Conversational Health Agents (CHAs), a prime example of this, are redefining healthcare by offering nuanced support that transcends textual analysis to incorporate emotional intelligence. This paper introduces an LLM-based CHA engineered for rich, multimodal dialogue-especially in the realm of mental health support. It adeptly interprets and responds to users' emotional states by analyzing multimodal cues, thus delivering contextually aware and empathetically resonant verbal responses. Our implementation leverages the versatile openCHA framework, and our comprehensive evaluation involves neutral prompts expressed in diverse emotional tones: sadness, anger, and joy. We evaluate the consistency and repeatability of the planning capability of the proposed CHA. Furthermore, human evaluators critique the CHA's empathic delivery, with findings revealing a striking concordance between the CHA's outputs and evaluators' assessments. These results affirm the indispensable role of vocal (soon multimodal) emotion recognition in strengthening the empathetic connection built by CHAs, cementing their place at the forefront of interactive, compassionate digital health solutions.
Abstract (translated)
代理是大型语言模型（LLMs）和生成式人工智能（GA）的一种最新兴的应用，其有效性取决于多模态能力来解决复杂的用户环境。会话健康代理（CHAs）是这种最著名的例子，通过提供 nuanced 的支持，超越了文本分析，并纳入了情商。本文介绍了一种基于LLM的CHA，特别适用于心理健康支持领域。它通过分析多模态线索来理解并回应用户的情绪状态，从而提供具有上下文感知和富有同情心的口头回应。我们的实现利用了灵活的openCHA框架，综合评估涉及多种情绪调度的中立提示。我们评估了所提出的CHA的规划能力的一致性和可重复性。此外，人类评估员批评了CHA的富有同情心的交付，这些研究结果揭示了CHA的输出与评估员评价之间令人印象深刻的一致性。这些结果证实了在加强CHAs所建立的同情连接方面，语音（很快将多模态）情感识别至关重要，巩固了它们在交互式、富有同情心的数字健康解决方案前沿的地位。
URL
https://arxiv.org/abs/2405.04777
