Abstract
Cued Speech (CS) is a visual communication system that combines lip-reading with hand coding to facilitate communication for individuals with hearing impairments. Automatic CS Recognition (ACSR) aims to convert CS hand gestures and lip movements into text via AI-driven methods. Traditionally, the temporal asynchrony between hand and lip movements requires the design of complex modules to facilitate effective multimodal fusion. However, constrained by limited data availability, current methods demonstrate insufficient capacity for adequately training these fusion mechanisms, resulting in suboptimal performance. Recently, multi-agent systems have shown promising capabilities in handling complex tasks with limited data availability. To this end, we propose the first collaborative multi-agent system for ACSR, named Cued-Agent. It integrates four specialized sub-agents: a Multimodal Large Language Model-based Hand Recognition agent that employs keyframe screening and CS expert prompt strategies to decode hand movements, a pretrained Transformer-based Lip Recognition agent that extracts lip features from the input video, a Hand Prompt Decoding agent that dynamically integrates hand prompts with lip features during inference in a training-free manner, and a Self-Correction Phoneme-to-Word agent that enables post-process and end-to-end conversion from phoneme sequences to natural language sentences for the first time through semantic refinement. To support this study, we expand the existing Mandarin CS dataset by collecting data from eight hearing-impaired cuers, establishing a mixed dataset of fourteen subjects. Extensive experiments demonstrate that our Cued-Agent performs superbly in both normal and hearing-impaired scenarios compared with state-of-the-art methods. The implementation is available at this https URL.
Abstract (translated)
Cued Speech（CS）是一种结合唇读和手码的视觉沟通系统，旨在为听力受损的人士提供有效的交流方式。自动化的Cued Speech识别（ACSR）目标是利用人工智能技术将手部动作和口型变化转换成文本信息。传统方法中，由于手势与口型间的时序不同步问题，需要设计复杂的模块来实现有效多模态融合。然而，受限于数据量的不足，现有方法在训练这些融合机制方面显得力不从心，导致性能不佳。 最近的研究显示，多智能体系统（multi-agent systems）在处理数据有限的任务中展现出巨大潜力。基于此，我们提出了一种用于ACSR的首个协作多智能体系统——Cued-Agent。该系统集成了四个专业化子代理：一个多模态大型语言模型驱动的手部识别代理，它通过关键帧筛选和CS专家提示策略解码手势；一个预训练的Transformer架构口型识别代理，从输入视频中提取口唇特征；一个手部提示解码代理，在推理过程中以无需训练的方式动态融合手部提示与口唇特征；以及一个自我纠错音素到单词转换代理，首次通过语义精炼实现从音素序列到自然语言句子的后处理和端到端转化。 为了支持这项研究，我们扩展了现有的普通话CS数据集，收集来自八位听力受损的Cued Speech用户的资料，并建立了一个由十四名用户组成的混合数据集。实验结果表明，我们的Cued-Agent在正常人和听力障碍场景下均优于现有最先进的方法。该系统的代码可在提供的链接中获取。
URL
https://arxiv.org/abs/2508.00391
