Abstract
The emergence of large-scale automatic speech recognition (ASR) models such as Whisper has greatly expanded their adoption across diverse real-world applications. Ensuring robustness against even minor input perturbations is therefore critical for maintaining reliable performance in real-time environments. While prior work has mainly examined accuracy degradation under adversarial attacks, robustness with respect to efficiency remains largely unexplored. This narrow focus provides only a partial understanding of ASR model vulnerabilities. To address this gap, we conduct a comprehensive study of ASR robustness under multiple attack scenarios. We introduce MORE, a multi-objective repetitive doubling encouragement attack, which jointly degrades recognition accuracy and inference efficiency through a hierarchical staged repulsion-anchoring mechanism. Specifically, we reformulate multi-objective adversarial optimization into a hierarchical framework that sequentially achieves the dual objectives. To further amplify effectiveness, we propose a novel repetitive encouragement doubling objective (REDO) that induces duplicative text generation by maintaining accuracy degradation and periodically doubling the predicted sequence length. Overall, MORE compels ASR models to produce incorrect transcriptions at a substantially higher computational cost, triggered by a single adversarial input. Experiments show that MORE consistently yields significantly longer transcriptions while maintaining high word error rates compared to existing baselines, underscoring its effectiveness in multi-objective adversarial attack.
Abstract (translated)
大规模自动语音识别（ASR）模型如Whisper的出现，极大地扩展了其在各种现实世界应用中的采用。因此，在面对轻微输入变化时确保模型鲁棒性对于保持实时环境下的可靠性能至关重要。尽管先前的工作主要研究了对抗攻击下准确率下降的问题，但关于效率方面的鲁棒性研究仍然相对较少。这种狭隘的关注点仅提供了对ASR模型脆弱性的部分理解。为了解决这一差距，我们进行了一项多场景攻击情况下的ASR模型鲁棒性全面研究。 为此，我们引入了MORE（多目标重复加倍鼓励攻击），该攻击通过分层阶段排斥-锚定机制同时降低识别准确率和推理效率。具体而言，我们将一个多目标对抗优化问题重新表述为一个逐步实现双重目标的层次框架。为了进一步放大效果，我们提出了一种新颖的重复鼓励加倍目标(REDO)，通过保持准确性下降的同时周期性地将预测序列长度加倍来诱导生成冗余文本。 总的来说，MORE促使ASR模型以更高的计算成本在单次对抗输入触发下产生错误转录。实验表明，与现有基线相比，MORE能够持续输出显著更长的转录，同时保持高词错率，这证明了其在多目标对抗攻击中的有效性。
URL
https://arxiv.org/abs/2601.01852
