Abstract
Automatic speech intelligibility assessment is crucial for monitoring speech disorders and therapy efficacy. However, existing methods are difficult to compare: research is fragmented across private datasets with inconsistent protocols. We introduce PathBench, a unified benchmark for pathological speech assessment using public datasets. We compare reference-free, reference-text, and reference-audio methods across three protocols (Matched Content, Extended, and Full) representing how a linguist (controlled stimuli) versus machine learning specialist (maximum data) would approach the same data. We establish benchmark baselines across six datasets, enabling systematic evaluation of future methodological advances, and introduce Dual-ASR Articulatory Precision (DArtP), achieving the highest average correlation among reference-free methods.
Abstract (translated)
自动语音清晰度评估对于监测言语障碍和治疗效果至关重要。然而，现有的方法难以进行比较：研究碎片化地分布在使用不一致协议的私人数据集上。我们引入了PathBench，这是一个统一基准，用于使用公共数据集进行病理语音评估。我们在三种不同的协议（匹配内容、扩展和完整）下对比了无参考、文本参考和音频参考的方法，这三种协议分别代表语言学家（控制刺激）与机器学习专家（最大数据量）在处理同一数据时的不同方法。我们建立了六个数据集上的基准线，从而能够对未来的研究进展进行系统的评估，并引入了一种新的指标——双ASR发音精确度（Dual-ASR Articulatory Precision, DArtP），该指标在无参考的方法中达到了最高的平均相关性。
URL
https://arxiv.org/abs/2603.08097
