Abstract
Video localization tasks aim to temporally locate specific instances in videos, including temporal action localization (TAL), sound event detection (SED) and audio-visual event localization (AVEL). Existing methods over-specialize on each task, overlooking the fact that these instances often occur in the same video to form the complete video content. In this work, we present UniAV, a Unified Audio-Visual perception network, to achieve joint learning of TAL, SED and AVEL tasks for the first time. UniAV can leverage diverse data available in task-specific datasets, allowing the model to learn and share mutually beneficial knowledge across tasks and modalities. To tackle the challenges posed by substantial variations in datasets (size/domain/duration) and distinct task characteristics, we propose to uniformly encode visual and audio modalities of all videos to derive generic representations, while also designing task-specific experts to capture unique knowledge for each task. Besides, we develop a unified language-aware classifier by utilizing a pre-trained text encoder, enabling the model to flexibly detect various types of instances and previously unseen ones by simply changing prompts during inference. UniAV outperforms its single-task counterparts by a large margin with fewer parameters, achieving on-par or superior performances compared to state-of-the-art task-specific methods across ActivityNet 1.3, DESED and UnAV-100 benchmarks.
Abstract (translated)
视频本地化任务旨在在视频内定位特定的实例，包括时间动作定位（TAL）、声音事件检测（SED）和音频-视觉事件定位（AVEL）。现有的方法过于专业化，忽视了这些实例通常在同一视频中发生，组成了完整的视频内容。在这项工作中，我们提出了UniAV，一个统一音频-视觉感知网络，实现了TAL、SED和AVEL任务的联合学习，这是第一次实现。UniAV可以利用任务特定数据集中的多样数据，使模型能够在任务和模式之间共享有益的知识。为了应对数据集（大小/领域/持续时间）的巨变和任务特征的差异，我们提出了统一编码所有视频的视觉和音频模态，以获得通用表示，同时为每个任务设计特定专家，捕捉独特知识。此外，我们还通过利用预训练的文本编码器，开发了一个统一语言感知的分类器，使模型在推理过程中可以灵活检测各种实例和之前未见过的实例，只需更改提示即可。UniAV在参数更少的情况下优于其单一任务 counterparts，在ActivityNet 1.3、DESED和UnAV-100基准测试中都实现了与最先进任务特定方法相当或更好的性能。
URL
https://arxiv.org/abs/2404.03179
