Abstract
Dysarthria is a motor speech disorder that results in slow and often incomprehensible speech. Speech intelligibility significantly impacts communication, leading to barriers in social interactions. Dysarthria is often a characteristic of neurological diseases including Parkinson's and ALS, yet current tools lack generalizability across languages and levels of severity. In this study, we present a unified AI-based multilingual framework that addresses six key components: (1) binary dysarthria detection, (2) severity classification, (3) clean speech generation, (4) speech-to-text conversion, (5) emotion detection, and (6) voice cloning. We analyze datasets in English, Russian, and German, using spectrogram-based visualizations and acoustic feature extraction to inform model training. Our binary detection model achieved 97% accuracy across all three languages, demonstrating strong generalization across languages. The severity classification model also reached 97% test accuracy, with interpretable results showing model attention focused on lower harmonics. Our translation pipeline, trained on paired Russian dysarthric and clean speech, reconstructed intelligible outputs with low training (0.03) and test (0.06) L1 losses. Given the limited availability of English dysarthric-clean pairs, we fine-tuned the Russian model on English data and achieved improved losses of 0.02 (train) and 0.03 (test), highlighting the promise of cross-lingual transfer learning for low-resource settings. Our speech-to-text pipeline achieved a Word Error Rate of 0.1367 after three epochs, indicating accurate transcription on dysarthric speech and enabling downstream emotion recognition and voice cloning from transcribed speech. Overall, the results and products of this study can be used to diagnose dysarthria and improve communication and understanding for patients across different languages.
Abstract (translated)
构音障碍是一种运动性言语障碍，导致缓慢且常难以理解的说话。言语清晰度对沟通有着重大影响，可能导致社交互动中的障碍。构音障碍常常是帕金森病和肌萎缩侧索硬化症等神经疾病的一个特征，然而目前的工具在跨语言和不同严重程度上缺乏普适性。 在这项研究中，我们提出了一种基于人工智能的多语种统一框架，涵盖六个关键组成部分：（1）构音障碍二元检测，（2）严重度分类，（3）清晰语音生成，（4）语音转文字转换，（5）情感识别和（6）声音克隆。我们在英语、俄语和德语的数据集上进行了分析，并使用基于频谱图的可视化以及声学特征提取来指导模型训练。 我们的二元检测模型在所有三种语言中均达到了97%的准确率，显示出良好的跨语言泛化能力。严重度分类模型也实现了97%的测试准确率，解释性结果表明模型注意力集中在较低谐波上。我们基于俄语构音障碍和清晰语音成对训练的语言翻译管道，在低训练（0.03）和测试（0.06）L1损失的情况下生成了可理解的输出。鉴于英语构音障碍-清晰对的数量有限，我们在英语数据集上微调了俄罗斯模型，并取得了改进后的训练（0.02）和测试（0.03）损失结果，这突显了跨语言迁移学习在资源匮乏环境中的潜力。 我们的语音转文字管道在三个周期后达到了0.1367的单词错误率，表明对构音障碍言语实现了准确转录，并使下游的情感识别和声音克隆成为可能。总体而言，本研究的结果和产品可用于诊断构音障碍并改善跨语言患者的沟通与理解。 这项研究表明了多语种统一框架在解决构音障碍相关问题上的潜力，特别是在资源有限的语言环境中，为提高患者的生活质量提供了新的方向和技术支持。
URL
https://arxiv.org/abs/2510.03986
