Abstract
Weakly supervised temporal action localization is a challenging task as only the video-level annotation is available during the training process. To address this problem, we propose a two-stage approach to fully exploit multi-resolution information in the temporal domain and generate high quality frame-level pseudo labels based on both appearance and motion streams. Specifically, in the first stage, we generate reliable initial frame-level pseudo labels, and in the second stage, we iteratively refine the pseudo labels and use a set of selected frames with highly confident pseudo labels to train neural networks and better predict action class scores at each frame. We fully exploit temporal information at multiple scales to improve temporal action localization performance. Specifically, in order to obtain reliable initial frame-level pseudo labels, in the first stage, we propose an Initial Label Generation (ILG) module, which leverages temporal multi-resolution consistency to generate high quality class activation sequences (CASs), which consist of a number of sequences with each sequence measuring how likely each video frame belongs to one specific action class. In the second stage, we propose a Progressive Temporal Label Refinement (PTLR) framework. In our PTLR framework, two networks called Network-OTS and Network-RTS, which are respectively used to generate CASs for the original temporal scale and the reduced temporal scales, are used as two streams (i.e., the OTS stream and the RTS stream) to refine the pseudo labels in turn. By this way, the multi-resolution information in the temporal domain is exchanged at the pseudo label level, and our work can help improve each stream (i.e., the OTS/RTS stream) by exploiting the refined pseudo labels from another stream (i.e., the RTS/OTS stream).
Abstract (translated)
弱监督时间动作定位是一项具有挑战性的任务，因为在训练过程中只有视频级别的注释可用。为了解决这个问题，我们提出了一种两阶段的方法来充分利用时域中的多分辨率信息，并基于外观和运动流生成高质量的帧级伪标签。具体来说，在第一阶段中，我们生成可靠的初始帧级伪标签；在第二阶段中，我们迭代地细化这些伪标签，并使用一组具有高置信度伪标签的选择性帧训练神经网络，以更好地预测每个帧的动作类别得分。 为了充分利用时域中的多分辨率信息来提高时间动作定位性能，我们在第一阶段提出了一个初始标签生成（ILG）模块。该模块利用了时间多分辨率一致性来生成高质量的类激活序列（CAS），这些序列由多个序列组成，每个序列衡量视频中每一帧属于特定动作类别的可能性。 在第二阶段，我们提出了一种渐进式时间标签细化（PTLR）框架，在此框架中使用两个称为Network-OTS和Network-RTS的网络分别生成原始时间和减少的时间尺度上的CAS。这两个网络作为两条流（即OTC流和RTC流），交替地用来细化伪标签。 通过这种方式，时域中的多分辨率信息以伪标签的形式进行交换，我们的工作可以通过利用另一条流中经过精炼的伪标签来改进每一条流（即OTC/RTC流）。
URL
https://arxiv.org/abs/2506.18261
