Abstract
Large audio-language models (LALMs) can generate reasoning chains for their predictions, but it remains unclear whether these reasoning chains remain grounded in the input audio. In this paper, we propose an RL-based strategy that grounds the reasoning outputs of LALMs with explicit timestamp annotations referring to relevant segments of the audio signal. Our analysis shows that timestamp grounding leads the model to attend more strongly to audio tokens during reasoning generation. Experiments on four speech-based benchmark datasets demonstrate that our approach improves performance compared to both zero-shot reasoning and fine-tuning without timestamp grounding. Additionally, grounding amplifies desirable reasoning behaviors, such as region exploration, audiology verification, and consistency, underscoring the importance of grounding mechanisms for faithful multimodal reasoning.
Abstract (translated)
大型音频语言模型（LALMs）能够为其预测生成推理链，但这些推理链是否始终扎根于输入音频仍不明确。本文提出一种基于强化学习的策略，通过显式的时间戳标注将LALMs的推理输出与音频信号的相关片段进行关联。我们的分析表明，时间戳关联使模型在生成推理过程中更加强烈地关注音频标记。在四个基于语音的基准数据集上的实验表明，与零样本推理和不含时间戳关联的微调相比，我们的方法提升了性能。此外，关联机制还增强了理想的推理行为，如区域探索、音频验证和一致性，这凸显了关联机制对于实现可靠的多模态推理至关重要。
URL
https://arxiv.org/abs/2603.19468
