Abstract
Weakly-supervised methods for video anomaly detection (VAD) are conventionally based merely on RGB spatio-temporal features, which continues to limit their reliability in real-world scenarios. This is due to the fact that RGB-features are not sufficiently distinctive in setting apart categories such as shoplifting from visually similar events. Therefore, towards robust complex real-world VAD, it is essential to augment RGB spatio-temporal features by additional modalities. Motivated by this, we introduce the Poly-modal Induced framework for VAD: "PI-VAD", a novel approach that augments RGB representations by five additional modalities. Specifically, the modalities include sensitivity to fine-grained motion (Pose), three dimensional scene and entity representation (Depth), surrounding objects (Panoptic masks), global motion (optical flow), as well as language cues (VLM). Each modality represents an axis of a polygon, streamlined to add salient cues to RGB. PI-VAD includes two plug-in modules, namely Pseudo-modality Generation module and Cross Modal Induction module, which generate modality-specific prototypical representation and, thereby, induce multi-modal information into RGB cues. These modules operate by performing anomaly-aware auxiliary tasks and necessitate five modality backbones -- only during training. Notably, PI-VAD achieves state-of-the-art accuracy on three prominent VAD datasets encompassing real-world scenarios, without requiring the computational overhead of five modality backbones at inference.
Abstract (translated)
弱监督视频异常检测（VAD）方法通常仅基于RGB时空特征，这在现实场景中继续限制了其可靠性。这是因为RGB特征不足以区分如偷窃等事件与视觉相似的其他事件。因此，在针对复杂真实世界的VAD问题时，增强RGB时空特征以包含额外模态是必要的。受到这一观点的启发，我们引入了一种新的方法——多模态诱导框架“PI-VAD”，它通过五种附加模态来增强RGB表示。具体来说，这些模态包括对细粒度运动（姿态）敏感性、三维场景和实体表示（深度）、周围物体（全景掩码）、全局运动（光流），以及语言线索（视觉语言模型）。每种模态在PI-VAD框架中代表一个多边形的一个轴，通过这种方式向RGB添加显著的提示信息。 PI-VAD包括两个插件模块：伪模态生成模块和跨模态诱导模块。这些模块分别生成特定于每个模态的原型表示，并将多模态信息引入到RGB线索中。这些模块操作时执行异常感知辅助任务，仅在训练期间需要五个模态骨干网络的支持。 值得注意的是，PI-VAD在涵盖真实世界场景的三个重要的VAD数据集上达到了最先进的准确率水平，且无需在推理阶段使用五种模态骨干网络来避免额外计算开销。
URL
https://arxiv.org/abs/2505.13123
