Abstract
Visual-Language Models (VLMs) have significantly advanced action video recognition. Supervised by the semantics of action labels, recent works adapt the visual branch of VLMs to learn video representations. Despite the effectiveness proved by these works, we believe that the potential of VLMs has yet to be fully harnessed. In light of this, we exploit the semantic units (SU) hiding behind the action labels and leverage their correlations with fine-grained items in frames for more accurate action recognition. SUs are entities extracted from the language descriptions of the entire action set, including body parts, objects, scenes, and motions. To further enhance the alignments between visual contents and the SUs, we introduce a multi-region module (MRA) to the visual branch of the VLM. The MRA allows the perception of region-aware visual features beyond the original global feature. Our method adaptively attends to and selects relevant SUs with visual features of frames. With a cross-modal decoder, the selected SUs serve to decode spatiotemporal video representations. In summary, the SUs as the medium can boost discriminative ability and transferability. Specifically, in fully-supervised learning, our method achieved 87.8\% top-1 accuracy on Kinetics-400. In K=2 few-shot experiments, our method surpassed the previous state-of-the-art by +7.1% and +15.0% on HMDB-51 and UCF-101, respectively.
Abstract (translated)
视觉语言模型(VLMs)已经极大地提高了动作视频识别的性能。通过执行动作标签语义,最近的工作调整了VLM的视觉分支以学习视频表示。尽管这些工作证明了VLM的潜力,但我们仍然相信VLM的潜力还没有完全 harness。因此,我们利用隐藏在动作标签背后的语义单元(SU)并利用它们与帧中的细粒度物品的相关性来更准确地执行动作识别。 SU 是从整个动作集合的语言描述中提取实体,包括身体部分、物体、场景和运动。为了进一步增强视觉内容和 SU 之间的对齐,我们引入了一个多区域模块(MRA)到 VLM 的视觉分支。 MRA 允许超越原始全局特征的区域感知视觉特征。我们的算法自适应地关注并选择与帧视觉特征相关的 SU。使用跨模态解码器,选定的 SU 用于解码时序视频表示。因此, SU 作为媒介可以增强区分能力和转移性。具体而言,在完全监督学习中,我们的方法在Kinetics-400上实现了87.8\%的第一名准确性。在 K=2的少量样本实验中,我们的方法分别比先前的最先进的技术提高了7.1%和15.0%。
URL
https://arxiv.org/abs/2303.09756
