Abstract
Effective communication is vital in healthcare, especially across language barriers, where non-verbal cues and gestures are critical. This paper presents a privacy-preserving vision-language framework for medical interpreter robots that detects specific speech acts (consent and instruction) and generates corresponding robotic gestures. Built on locally deployed open-source models, the system utilizes a Large Language Model (LLM) with few-shot prompting for intent detection. We also introduce a novel dataset of clinical conversations annotated for speech acts and paired with gesture clips. Our identification module achieved 0.90 accuracy, 0.93 weighted precision, and a 0.91 weighted F1-Score. Our approach significantly improves computational efficiency and, in user studies, outperforms the speech-gesture generation baseline in human-likeness while maintaining comparable appropriateness.
Abstract (translated)
有效的沟通在医疗保健中至关重要，尤其是在语言障碍的情况下，非言语线索和手势在此时尤为重要。本文介绍了一种隐私保护的视觉-语言框架，用于医学翻译机器人，该框架能够检测特定的语言行为（如同意和指示）并生成相应的机器人动作。此系统基于本地部署的开源模型构建，并使用少量样本提示的大规模语言模型（LLM）进行意图识别。我们还引入了一个新的临床对话数据集，其中包含了针对言语行为标注的数据以及对应的肢体动作片段。 我们的身份识别模块达到了0.90的准确性、0.93加权精确度和0.91加权F1值。我们的方法显著提高了计算效率，并且在用户研究中，与现有的基于语音-手势生成的方法相比，在人机相似度方面表现更佳，同时保持了相当的适当性。 这段文字总结了一种用于医疗翻译机器人的系统框架及其性能特征，旨在通过有效的视觉和语言交互来克服语言障碍。
URL
https://arxiv.org/abs/2603.05751
