Abstract
Taiwanese Hakka is a low-resource, endangered language that poses significant challenges for automatic speech recognition (ASR), including high dialectal variability and the presence of two distinct writing systems (Hanzi and Pinyin). Traditional ASR models often encounter difficulties in this context, as they tend to conflate essential linguistic content with dialect-specific variations across both phonological and lexical dimensions. To address these challenges, we propose a unified framework grounded in the Recurrent Neural Network Transducers (RNN-T). Central to our approach is the introduction of dialect-aware modeling strategies designed to disentangle dialectal "style" from linguistic "content", which enhances the model's capacity to learn robust and generalized representations. Additionally, the framework employs parameter-efficient prediction networks to concurrently model ASR (Hanzi and Pinyin). We demonstrate that these tasks create a powerful synergy, wherein the cross-script objective serves as a mutual regularizer to improve the primary ASR tasks. Experiments conducted on the HAT corpus reveal that our model achieves 57.00% and 40.41% relative error rate reduction on Hanzi and Pinyin ASR, respectively. To our knowledge, this is the first systematic investigation into the impact of Hakka dialectal variations on ASR and the first single model capable of jointly addressing these tasks.
Abstract (translated)
台湾客家话是一种资源较少且濒临灭绝的语言，给自动语音识别（ASR）带来了重大挑战，包括高度的方言变异性以及两种不同的书写系统（汉字和拼音）。传统的ASR模型在这种情况下往往遇到困难，因为它们倾向于混淆语言内容与特定于方言的变化，这些变化在音韵和词汇维度上都有体现。为了应对这些挑战，我们提出了一种基于循环神经网络转换器（RNN-T）的统一框架。我们的方法核心在于引入了意识方言建模策略，旨在将方言特有的“风格”从语言通用的“内容”中分离出来，从而增强模型学习稳健且泛化的表示能力。此外，该框架还采用参数高效的预测网络来同时处理ASR任务（汉字和拼音）。我们证明，这些任务之间存在着强大的协同效应，即跨书写系统的目标可以作为相互的正则化器以提升主要的ASR任务性能。 在HAT语料库上进行的实验表明，我们的模型分别实现了57.00% 和40.41% 的相对错误率降低，针对汉字和拼音ASR。据我们所知，这是首次系统地研究客家方言变异性对ASR的影响，并且是首个能够同时处理这些任务的单一模型。
URL
https://arxiv.org/abs/2602.22522
