Abstract
Word error rate (WER) is a metric used to evaluate the quality of transcriptions produced by Automatic Speech Recognition (ASR) systems. In many applications, it is of interest to estimate WER given a pair of a speech utterance and a transcript. Previous work on WER estimation focused on building models that are trained with a specific ASR system in mind (referred to as ASR system-dependent). These are also domain-dependent and inflexible in real-world applications. In this paper, a hypothesis generation method for ASR System-Independent WER estimation (SIWE) is proposed. In contrast to prior work, the WER estimators are trained using data that simulates ASR system output. Hypotheses are generated using phonetically similar or linguistically more likely alternative words. In WER estimation experiments, the proposed method reaches a similar performance to ASR system-dependent WER estimators on in-domain data and achieves state-of-the-art performance on out-of-domain data. On the out-of-domain data, the SIWE model outperformed the baseline estimators in root mean square error and Pearson correlation coefficient by relative 17.58% and 18.21%, respectively, on Switchboard and CALLHOME. The performance was further improved when the WER of the training set was close to the WER of the evaluation dataset.
Abstract (translated)
词错误率（WER）是一种用于评估自动语音识别（ASR）系统产生的转录质量的度量。在许多应用中，估计WER根据语音单位和文本的关系进行估计。之前的工作主要集中在构建特定ASR系统的模型（称为ASR系统依赖的模型）。这些模型在领域上是相关的，且在现实世界的应用中不灵活。在本文中，提出了一个用于ASR System-Independent WER估计（SIWE）的假设生成方法。与之前的工作不同，使用模拟ASR系统输出的数据来训练WER估计算法。假设是通过 phonetically similar 或 linguistically more likely 替代词来生成的。在WER估计实验中，所提出的方法在领域内数据上的性能与ASR系统依赖的WER估计算法相当，同时在对外部数据上的性能达到了最先进的水平。在外部数据上，SIWE模型分别比基线估计器在Switchboard和CALLHOME上的相对误差的绝对值和皮尔逊相关系数上提高了17.58%和18.21%。当训练集的WER接近评估数据上的WER时，性能进一步得到改善。
URL
https://arxiv.org/abs/2404.16743
