Abstract
Weakly-supervised temporal action localization (WTAL) aims to recognize and localize action instances with only video-level labels. Despite the significant progress, existing methods suffer from severe performance degradation when transferring to different distributions and thus may hardly adapt to real-world scenarios . To address this problem, we propose the Generalizable Temporal Action Localization task (GTAL), which focuses on improving the generalizability of action localization methods. We observed that the performance decline can be primarily attributed to the lack of generalizability to different action scales. To address this problem, we propose STAT (Self-supervised Temporal Adaptive Teacher), which leverages a teacher-student structure for iterative refinement. Our STAT features a refinement module and an alignment module. The former iteratively refines the model's output by leveraging contextual information and helps adapt to the target scale. The latter improves the refinement process by promoting a consensus between student and teacher models. We conduct extensive experiments on three datasets, THUMOS14, ActivityNet1.2, and HACS, and the results show that our method significantly improves the Baseline methods under the cross-distribution evaluation setting, even approaching the same-distribution evaluation performance.
Abstract (translated)
弱监督的时间动作定位（WTAL）旨在仅通过视频级别的标签来识别和定位动作实例。尽管取得了显著的进展，但现有的方法在转移到不同分布时性能严重下降，因此可能难以适应现实场景。为解决这个问题，我们提出了泛化的动作定位任务（GTAL），它关注于提高动作定位方法的泛化能力。我们观察到，性能下降主要归因于不同动作规模缺乏泛化能力。为解决这个问题，我们提出了STAT（自监督的时间适应教师），它利用了教师-学生结构进行迭代精炼。我们的STAT包括精炼模块和调整模块。前者通过利用上下文信息逐步优化模型的输出，有助于适应目标规模。后者通过促进学生和教师模型的共识来改善精炼过程。我们在THUMOS14、ActivityNet1.2和HACS三个数据集上进行了广泛的实验，结果表明，我们的方法在跨分布评估设置下显著提高了基线方法的成绩，甚至接近同分布评估的性能。
URL
https://arxiv.org/abs/2404.13311
