Abstract
Temporal Action Localization (TAL) involves localizing and classifying action snippets in an untrimmed video. The emergence of large video foundation models has led RGB-only video backbones to outperform previous methods needing both RGB and optical flow modalities. Leveraging these large models is often limited to training only the TAL head due to the prohibitively large GPU memory required to adapt the video backbone for TAL. To overcome this limitation, we introduce LoSA, the first memory-and-parameter-efficient backbone adapter designed specifically for TAL to handle untrimmed videos. LoSA specializes for TAL by introducing Long-Short-range Adapters that adapt the intermediate layers of the video backbone over different temporal ranges. These adapters run parallel to the video backbone to significantly reduce memory footprint. LoSA also includes Long-Short-range Fusion that strategically combines the output of these adapters from the video backbone layers to enhance the video features provided to the TAL head. Experiments show that LoSA significantly outperforms all existing methods on standard TAL benchmarks, THUMOS-14 and ActivityNet-v1.3, by scaling end-to-end backbone adaptation to billion-parameter-plus models like VideoMAEv2~(ViT-g) and leveraging them beyond head-only transfer learning.
Abstract (translated)
Temporal Action Localization（TAL）涉及在未剪辑的视频中定位和分类动作片段。大型视频基础模型的发展使得仅使用RGB视频骨干的先前方法已经无法比需要同时具备RGB和光学流模态的先前方法更优秀。利用这些大型模型通常局限于仅训练TAL头部，因为需要横跨GPU内存训练视频骨干。为了克服这一限制，我们引入了LoSA，专为TAL设计的第一个内存和参数高效的骨架适配器，以处理未剪辑的视频。LoSA专门为TAL设计，通过引入长-短程适配器来调整视频骨干的中间层。这些适配器与视频骨干并行运行，显著减少了内存足迹。LoSA还包括长-短程融合，将来自视频骨干层的中输出适配器策略性地组合以增强TAL头提供的视频特征。实验证明，LoSA在标准TAL基准测试、THUMOS-14和ActivityNet-v1.3等所有现有方法中均显著胜出，通过将端到端骨架适应性扩展到像VideoMAEv2~（ViT-g）这样的大规模参数模型，并超越仅头部传输学习。
URL
https://arxiv.org/abs/2404.01282
