Abstract
Recently, the Large Language Model-based Phoneme-to-Grapheme (LLM-P2G) method has shown excellent performance in speech recognition tasks and has become a feasible direction to replace the traditional WFST decoding method. This framework takes into account both recognition accuracy and system scalability through two-stage modeling of phoneme prediction and text generation. However, the existing LLM-P2G adopts the Top-K Marginalized (TKM) training strategy, and its candidate phoneme sequences rely on beam search generation, which has problems such as insufficient path diversity, low training efficiency, and high resource overhead. To this end, this paper proposes a sampling marginalized training strategy (Sampling-K Marginalized, SKM), which replaces beam search with random sampling to generate candidate paths, improving marginalized modeling and training efficiency. Experiments were conducted on Polish and German datasets, and the results showed that SKM further improved the model learning convergence speed and recognition performance while maintaining the complexity of the model. Comparative experiments with a speech recognition method that uses a projector combined with a large language model (SpeechLLM) also show that the SKM-driven LLM-P2G has more advantages in recognition accuracy and structural simplicity. The study verified the practical value and application potential of this method in cross-language speech recognition systems.
Abstract (translated)
最近，基于大型语言模型的音素到字母（LLM-P2G）方法在语音识别任务中表现出色，并已成为取代传统WFST解码方法的一个可行方向。该框架通过音素预测和文本生成的两阶段建模来兼顾识别准确性和系统可扩展性。然而，现有的LLM-P2G采用的是Top-K边际化（TKM）训练策略，其候选音素序列依赖于束搜索生成，存在路径多样性不足、训练效率低以及资源消耗高的问题。 为此，本文提出了一种采样边际化训练策略（Sampling-K Marginalized, SKM），该策略用随机抽样代替束搜索来生成候选路径，从而改进了边际化建模，并提高了训练效率。在波兰语和德语数据集上的实验表明，SKM进一步加快了模型的学习收敛速度并提升了识别性能，同时保持了模型的复杂度不变。与结合投影仪和大型语言模型（SpeechLLM）的语音识别方法进行比较实验的结果也显示，由SKM驱动的LLM-P2G在识别准确性和结构简洁性方面具有更多优势。这项研究验证了该方法在跨语言语音识别系统中的实际价值和应用潜力。
URL
https://arxiv.org/abs/2512.18371
