Abstract
Large Audio-Language Models (LALMs) typically struggle with localized dialectal prosody due to the scarcity of specialized corpora. We present TW-Sound580K, a Taiwanese audio-text instruction dataset developed through a Verify-Generate-Critique (VGC) protocol. This pipeline leverages Dual-ASR validation to filter 522K raw clips, subsequently expanding them into 580,000 high-fidelity instruction pairs using a teacher model. The dataset's utility is demonstrated through Tai-LALM, which fine-tunes a DeSTA 2.5-Audio-initialized backbone and incorporates a dynamic Dual-ASR Arbitration strategy to optimize transcription selection during inference. On the TAU Benchmark, Tai-LALM reaches 49.1% accuracy, marking a 6.5% absolute improvement over the zero-shot baseline (42.6% with ASR text conditioning). This confirms that integrating regional corpora with rigorous curation and dynamic arbitration significantly enhances LALM performance on localized speech.
Abstract (translated)
大型音频-语言模型（LALMs）通常在处理地方方言韵律方面遇到困难，这是因为缺乏专门的语料库。我们提出了TW-Sound580K，这是一个通过验证-生成-批评(VGC)协议开发的台湾音频-文本指令数据集。这一流程利用双ASR验证来筛选出52.2万个原始片段，并随后使用教师模型将其扩展为58万个高质量的指令对。 该数据集的应用价值通过Tai-LALM得到了展示，即通过对DeSTA 2.5-Audio初始化的骨干网络进行微调并采用动态双ASR仲裁策略，在推理过程中优化转录选择。在TAU基准测试中，Tai-LALM达到了49.1%的准确率，比零样本基线（使用ASR文本条件为42.6%）提高了6.5个百分点的绝对值。这证实了整合具有严格策划和动态仲裁的地方语料库能够显著提高LALM在地方口语处理上的表现。
URL
https://arxiv.org/abs/2603.05094
