Abstract
Test-time adaptation (TTA) aims to boost the generalization capability of a trained model by conducting self-/unsupervised learning during the testing phase. While most existing TTA methods for video primarily utilize visual supervisory signals, they often overlook the potential contribution of inherent audio data. To address this gap, we propose a novel approach that incorporates audio information into video TTA. Our method capitalizes on the rich semantic content of audio to generate audio-assisted pseudo-labels, a new concept in the context of video TTA. Specifically, we propose an audio-to-video label mapping method by first employing pre-trained audio models to classify audio signals extracted from videos and then mapping the audio-based predictions to video label spaces through large language models, thereby establishing a connection between the audio categories and video labels. To effectively leverage the generated pseudo-labels, we present a flexible adaptation cycle that determines the optimal number of adaptation iterations for each sample, based on changes in loss and consistency across different views. This enables a customized adaptation process for each sample. Experimental results on two widely used datasets (UCF101-C and Kinetics-Sounds-C), as well as on two newly constructed audio-video TTA datasets (AVE-C and AVMIT-C) with various corruption types, demonstrate the superiority of our approach. Our method consistently improves adaptation performance across different video classification models and represents a significant step forward in integrating audio information into video TTA. Code: this https URL.
Abstract (translated)
测试时适应（TTA，Test-time Adaptation）旨在通过在测试阶段进行自监督或无监督学习来增强训练模型的泛化能力。尽管现有的大多数视频TTA方法主要利用视觉线索，但它们通常忽视了音频数据固有的潜在贡献。为了解决这一问题，我们提出了一种新方法，将音频信息融入到视频TTA中。我们的方法利用音频中的丰富语义内容生成辅助伪标签，这是视频TTA领域的一个新概念。具体来说，我们提出了一个从音频到视频的标签映射方法：首先使用预训练的音频模型对提取自视频的音频信号进行分类，然后通过大型语言模型将基于音频的预测映射至视频标签空间中，从而建立起音频类别与视频标签之间的联系。 为了有效地利用生成的伪标签，我们提出了一种灵活适应循环，可以根据损失和不同视角的一致性变化来确定每个样本的最佳适应迭代次数。这使得可以为每个样本定制一个适应过程。在两个常用的基准数据集（UCF101-C和Kinetics-Sounds-C）以及两个新的音频-视频TTA数据集（AVE-C和AVMIT-C）上进行的实验，表明了我们的方法对各种损坏类型的有效性。 实验结果表明，在不同的视频分类模型中，我们的方法始终提升适应性能，并且在将音频信息整合到视频TTA方面取得了重要进展。代码：[提供链接]。
URL
https://arxiv.org/abs/2506.12481
