Abstract
Understanding videos that contain multiple modalities is crucial, especially in egocentric videos, where combining various sensory inputs significantly improves tasks like action recognition and moment localization. However, real-world applications often face challenges with incomplete modalities due to privacy concerns, efficiency needs, or hardware issues. Current methods, while effective, often necessitate retraining the model entirely to handle missing modalities, making them computationally intensive, particularly with large training datasets. In this study, we propose a novel approach to address this issue at test time without requiring retraining. We frame the problem as a test-time adaptation task, where the model adjusts to the available unlabeled data at test time. Our method, MiDl~(Mutual information with self-Distillation), encourages the model to be insensitive to the specific modality source present during testing by minimizing the mutual information between the prediction and the available modality. Additionally, we incorporate self-distillation to maintain the model's original performance when both modalities are available. MiDl represents the first self-supervised, online solution for handling missing modalities exclusively at test time. Through experiments with various pretrained models and datasets, MiDl demonstrates substantial performance improvement without the need for retraining.
Abstract (translated)
理解包含多种模态的视频对来说非常重要，尤其是在自闭型视频中，将各种感官输入结合起来可以显著提高诸如动作识别和时刻定位等任务。然而，由于隐私问题、效率需求或硬件问题等原因，现实世界的应用经常面临模态不完整的情况。尽管现有的方法非常有效，但通常需要重新训练整个模型来处理缺失的模态，这使得它们在计算上是密集的，尤其是在大型训练数据集的情况下。在本文中，我们提出了一种在测试时不需要重新训练的方法来解决这个问题。我们将问题建模为测试时的自适应任务，在这个任务中，模型根据测试时的未标注数据进行调整。我们的方法MiDl~（ mutual information with self-distillation）通过最小化预测和可用模态之间的互信息来鼓励模型对测试时的具体模态保持鲁棒性。此外，我们还将自监督学习集成到模型中，以便在模态都存在时保持模型的原始性能。MiDl是第一个在测试时专门处理缺失模态的自监督在线解决方案。通过使用各种预训练模型和数据集进行实验，MiDl证明了在无需重新训练的情况下具有显著的性能提升。
URL
https://arxiv.org/abs/2404.15161
