Abstract
Referring Atomic Video Action Recognition (RAVAR) aims to recognize fine-grained, atomic-level actions of a specific person of interest conditioned on natural language descriptions. Distinct from conventional action recognition and detection tasks, RAVAR emphasizes precise language-guided action understanding, which is particularly critical for interactive human action analysis in complex multi-person scenarios. In this work, we extend our previously introduced RefAVA dataset to RefAVA++, which comprises >2.9 million frames and >75.1k annotated persons in total. We benchmark this dataset using baselines from multiple related domains, including atomic action localization, video question answering, and text-video retrieval, as well as our earlier model, RefAtomNet. Although RefAtomNet surpasses other baselines by incorporating agent attention to highlight salient features, its ability to align and retrieve cross-modal information remains limited, leading to suboptimal performance in localizing the target person and predicting fine-grained actions. To overcome the aforementioned limitations, we introduce RefAtomNet++, a novel framework that advances cross-modal token aggregation through a multi-hierarchical semantic-aligned cross-attention mechanism combined with multi-trajectory Mamba modeling at the partial-keyword, scene-attribute, and holistic-sentence levels. In particular, scanning trajectories are constructed by dynamically selecting the nearest visual spatial tokens at each timestep for both partial-keyword and scene-attribute levels. Moreover, we design a multi-hierarchical semantic-aligned cross-attention strategy, enabling more effective aggregation of spatial and temporal tokens across different semantic hierarchies. Experiments show that RefAtomNet++ establishes new state-of-the-art results. The dataset and code are released at this https URL.
Abstract (translated)
基于参考的原子视频动作识别（RAVAR）的目标是在自然语言描述的基础上，对特定感兴趣人物的细微、原子级的动作进行识别。与传统的动作识别和检测任务不同，RAVAR强调精准的语言引导下的动作理解，在复杂的多人场景中互动的人类动作分析尤为重要。 在这一工作中，我们扩展了先前介绍的RefAVA数据集到RefAVA++版本，该数据集包括超过290万帧和总计75,100多个标注人物。我们在这一数据集上使用来自多个相关领域的基线模型进行了基准测试，这些领域包括原子动作定位、视频问答以及文本-视频检索，并且还包含了我们早期开发的RefAtomNet模型。 尽管RefAtomNet通过引入代理人注意机制突出显示显著特征从而超越了其他基线模型，但其在跨模态信息对齐与检索方面的能力仍然有限，这导致它在定位目标人物和预测细微动作时表现不佳。为克服上述限制，我们提出了RefAtomNet++这一新型框架，该框架通过引入多层次语义对齐的跨注意力机制以及多轨迹Mamba建模（分别针对部分关键词、场景属性和整体句子级别）进一步推进了跨模态标记聚合。 具体而言，在部分关键词和场景属性水平上，扫描轨迹是通过对每个时间步长动态选择最近的视觉空间令牌来构建。此外，我们设计了一个多层次语义对齐的跨注意力策略，使得在不同的语义层次之间更有效地聚集时空令牌成为可能。实验表明，RefAtomNet++确立了新的最先进的结果。 数据集和代码已在此链接处发布。
URL
https://arxiv.org/abs/2510.16444
