Abstract
We introduce a new task called Referring Atomic Video Action Recognition (RAVAR), aimed at identifying atomic actions of a particular person based on a textual description and the video data of this person. This task differs from traditional action recognition and localization, where predictions are delivered for all present individuals. In contrast, we focus on recognizing the correct atomic action of a specific individual, guided by text. To explore this task, we present the RefAVA dataset, containing 36,630 instances with manually annotated textual descriptions of the individuals. To establish a strong initial benchmark, we implement and validate baselines from various domains, e.g., atomic action localization, video question answering, and text-video retrieval. Since these existing methods underperform on RAVAR, we introduce RefAtomNet -- a novel cross-stream attention-driven method specialized for the unique challenges of RAVAR: the need to interpret a textual referring expression for the targeted individual, utilize this reference to guide the spatial localization and harvest the prediction of the atomic actions for the referring person. The key ingredients are: (1) a multi-stream architecture that connects video, text, and a new location-semantic stream, and (2) cross-stream agent attention fusion and agent token fusion which amplify the most relevant information across these streams and consistently surpasses standard attention-based fusion on RAVAR. Extensive experiments demonstrate the effectiveness of RefAtomNet and its building blocks for recognizing the action of the described individual. The dataset and code will be made publicly available at this https URL.
Abstract (translated)
我们提出了一个名为 Referring Atomic Video Action Recognition (RAVAR) 的新任务，旨在根据特定人员的文本描述和视频数据识别该人员的原子动作。与传统的动作识别和定位不同，我们的任务专注于识别特定人员的正确原子动作，通过文本指导。为了探索这个任务，我们提出了 RefAVA 数据集，其中包括 36,630 个带有手动标注的个体文本描述的实例。为了建立一个强大的初始基准，我们从各种领域实现了各种基线，例如原子动作的定位、视频问题回答和文本-视频检索。由于这些现有方法在 RAVAR 上表现不佳，我们引入了 RefAtomNet——一个专门针对 RAVAR 独特挑战的新颖跨流关注方法：为指定人员解释文本指征，利用这个参考引导空间定位并捕获指定期人的原子动作预测。关键成分是：（1）连接视频、文本和新位置语义流的Multi-stream架构；（2）跨流代理注意力和代理标记融合，增强这些通道中的最相关信息，并 consistently超越基于标准的注意力融合在 RAVAR 上。大量的实验证明 RefAtomNet 和其构建模块在识别描述人员的动作方面非常有效。数据集和代码将公开发布在这个 https URL 上。
URL
https://arxiv.org/abs/2407.01872
