Abstract
Audio-Visual Speech Recognition (AVSR) leverages both audio and visual modalities to enhance speech recognition robustness, particularly in noisy environments. Recent advancements in Large Language Models (LLMs) have demonstrated their effectiveness in speech recognition, including AVSR. However, due to the significant length of speech representations, direct integration with LLMs imposes substantial computational costs. Prior approaches address this by compressing speech representations before feeding them into LLMs. However, higher compression ratios often lead to performance degradation, necessitating a trade-off between computational efficiency and recognition accuracy. To address this challenge, we propose Llama-MTSK, the first Matryoshka-based Multimodal LLM for AVSR, which enables flexible adaptation of the audio-visual token allocation based on specific computational constraints while preserving high performance. Our approach, inspired by Matryoshka Representation Learning, encodes audio-visual representations at multiple granularities within a single model, eliminating the need to train separate models for different compression levels. Moreover, to efficiently fine-tune the LLM, we introduce three LoRA-based Matryoshka strategies using global and scale-specific LoRA modules. Extensive evaluations on the two largest AVSR datasets demonstrate that Llama-MTSK achieves state-of-the-art results, matching or surpassing models trained independently at fixed compression levels.
Abstract (translated)
音频-视觉语音识别（AVSR）利用声音和视觉两种模式来增强语音识别的鲁棒性，特别是在嘈杂环境中。最近，在大型语言模型（LLMs）在包括AVSR在内的语音识别方面的有效性得到了展示。然而，由于语音表示的长度较长，直接与LLMs集成会带来高昂的计算成本。先前的方法通过压缩语音表示后再将其输入到LLMs中来解决这一问题。但是，较高的压缩比率通常会导致性能下降，这需要在计算效率和识别准确性之间做出权衡。 为了解决这个问题，我们提出了Llama-MTSK——第一个基于Matryoshka原理的多模态大语言模型，用于AVSR任务。它能够根据特定的计算约束灵活地调整音频-视觉标记分配，同时保持高性能。我们的方法受到Matryoshka表征学习的启发，在单一模型内以多种粒度级别编码音频-视觉表示，从而无需为不同压缩水平训练单独的模型。 此外，为了高效地微调LLM，我们引入了三种基于LoRA（低秩适应）的方法：使用全局和特定尺度的LoRA模块。在两个最大的AVSR数据集上进行的广泛评估显示，Llama-MTSK实现了最先进的结果，其性能与固定压缩水平下独立训练的模型相匹配或超越。 总之，我们的研究展示了一种创新方法，通过灵活调整音频-视觉标记分配和采用先进的微调策略来应对计算效率和识别准确性之间的挑战。这为未来的AVSR任务提供了新的方向。
URL
https://arxiv.org/abs/2503.06362
