Abstract
Zero-Shot Temporal Action Localization (ZS-TAL) seeks to identify and locate actions in untrimmed videos unseen during training. Existing ZS-TAL methods involve fine-tuning a model on a large amount of annotated training data. While effective, training-based ZS-TAL approaches assume the availability of labeled data for supervised learning, which can be impractical in some applications. Furthermore, the training process naturally induces a domain bias into the learned model, which may adversely affect the model's generalization ability to arbitrary videos. These considerations prompt us to approach the ZS-TAL problem from a radically novel perspective, relaxing the requirement for training data. To this aim, we introduce a novel method that performs Test-Time adaptation for Temporal Action Localization (T3AL). In a nutshell, T3AL adapts a pre-trained Vision and Language Model (VLM). T3AL operates in three steps. First, a video-level pseudo-label of the action category is computed by aggregating information from the entire video. Then, action localization is performed adopting a novel procedure inspired by self-supervised learning. Finally, frame-level textual descriptions extracted with a state-of-the-art captioning model are employed for refining the action region proposals. We validate the effectiveness of T3AL by conducting experiments on the THUMOS14 and the ActivityNet-v1.3 datasets. Our results demonstrate that T3AL significantly outperforms zero-shot baselines based on state-of-the-art VLMs, confirming the benefit of a test-time adaptation approach.
Abstract (translated)
Zero-Shot Temporal Action Localization (ZS-TAL)旨在识别和定位未在训练过程中见过的视频中的动作。现有的ZS-TAL方法需要在大量注释训练数据上对模型进行微调。虽然有效，但基于训练的ZS-TAL方法假设存在用于监督学习的有标注数据，这在某些应用中可能不可行。此外，训练过程会自然地引入领域偏见到学习到的模型中，这可能会影响模型对任意视频的泛化能力。这些考虑促使我们从一种全新且新颖的角度来解决ZS-TAL问题，放宽对训练数据的要求。为此，我们引入了一种名为T3AL的新方法，用于对Temporal Action Localization (T3AL)进行测试时间适应。 T3AL的工作原理如下。首先，通过汇总整个视频的信息，计算出动作类别的视频级伪标签。然后，采用一种新型的方法，受到自监督学习启发的动作定位。最后，采用最先进的捕捉模型的帧级文本描述来微调动作区域建议。 我们对THUMOS14和ActivityNet-v1.3数据集进行了实验验证。结果表明，T3AL在基于最佳VLMs的零散拍摄基准上显著优于零散拍摄基线。这证实了测试时间适应方法的优势。
URL
https://arxiv.org/abs/2404.05426
