Abstract
There has been increasing interest in unifying streaming and non-streaming automatic speech recognition (ASR) models to reduce development, training, and deployment costs. We present a unified framework that trains a single end-to-end ASR model for both streaming and non-streaming applications, leveraging future context information. We propose to use dynamic right-context through the chunked attention masking in the training of zipformer-based ASR models. We demonstrate that using right-context is more effective in zipformer models compared to other conformer models due to its multi-scale nature. We analyze the effect of varying the number of right-context frames on accuracy and latency of the streaming ASR models. We use Librispeech and large in-house conversational datasets to train different versions of streaming and non-streaming models and evaluate them in a production grade server-client setup across diverse testsets of different domains. The proposed strategy reduces word error by relative 7.9\% with a small degradation in user-perceived latency. By adding more right-context frames, we are able to achieve streaming performance close to that of non-streaming models. Our approach also allows flexible control of the latency-accuracy tradeoff according to customers requirements.
Abstract (translated)
人们对统一流式和非流式自动语音识别（ASR）模型的兴趣日益增加，以减少开发、训练和部署成本。我们提出了一种统一的框架，该框架使用单个端到端ASR模型同时为流式和非流式应用进行训练，并利用未来上下文信息。我们建议通过在基于zipformer的ASR模型中采用分块注意掩码来动态使用右上下文（right-context）。我们展示了与其它conformer模型相比，由于其多尺度特性，在zipformer模型中使用右上下文更为有效。我们分析了不同数量的右上下文帧对流式ASR模型准确率和延迟的影响，并使用Librispeech及大规模内部对话数据集来训练各种版本的流式和非流式模型，并在跨多个域的不同测试集中进行生产级服务器-客户端设置下的评估。所提出的策略使单词错误相对减少了7.9%，同时用户感知的延迟略有下降。通过增加更多的右上下文帧，我们可以实现接近非流式模型性能的流式表现。此外，我们的方法还允许根据客户需求灵活控制延迟和准确率之间的权衡。
URL
https://arxiv.org/abs/2506.14434
