Abstract
With the increasing adoption of video anomaly detection in intelligent surveillance domains, conventional visual-based detection approaches often struggle with information insufficiency and high false-positive rates in complex environments. To address these limitations, we present a novel weakly supervised framework that leverages audio-visual collaboration for robust video anomaly detection. Capitalizing on the exceptional cross-modal representation learning capabilities of Contrastive Language-Image Pretraining (CLIP) across visual, audio, and textual domains, our framework introduces two major innovations: an efficient audio-visual fusion that enables adaptive cross-modal integration through lightweight parametric adaptation while maintaining the frozen CLIP backbone, and a novel audio-visual prompt that dynamically enhances text embeddings with key multimodal information based on the semantic correlation between audio-visual features and textual labels, significantly improving CLIP's generalization for the video anomaly detection task. Moreover, to enhance robustness against modality deficiency during inference, we further develop an uncertainty-driven feature distillation module that synthesizes audio-visual representations from visual-only inputs. This module employs uncertainty modeling based on the diversity of audio-visual features to dynamically emphasize challenging features during the distillation process. Our framework demonstrates superior performance across multiple benchmarks, with audio integration significantly boosting anomaly detection accuracy in various scenarios. Notably, with unimodal data enhanced by uncertainty-driven distillation, our approach consistently outperforms current unimodal VAD methods.
Abstract (translated)
随着视频异常检测在智能监控领域中的广泛应用，传统的基于视觉的方法常常在复杂环境中面临信息不足和高误报率的问题。为了解决这些问题，我们提出了一种新的弱监督框架，该框架利用音频-视觉协作进行鲁棒的视频异常检测。我们的框架借鉴了对比语言图像预训练（CLIP）在视觉、音频和文本领域中出色的跨模态表示学习能力，并引入了两项主要创新：一种高效的音频-视觉融合方法，它通过轻量级参数调整实现自适应跨模式整合，同时保持CLIP骨干网络不变；以及一种新颖的音频-视觉提示机制，该机制根据音频-视觉特征与文本标签之间的语义关联动态增强文本嵌入中的关键多模态信息，显著提高了CLIP在视频异常检测任务中的泛化能力。 此外，为了提高在推理过程中对模态不足的鲁棒性，我们进一步开发了一个基于不确定性的特征蒸馏模块，该模块可以从仅有的视觉输入中综合音频-视觉表示。此模块采用基于音频-视觉特性多样性的不确定性建模方法，在蒸馏过程中动态强调具有挑战性的特征。 我们的框架在多个基准测试中的性能表现出色，尤其是在各种场景下，通过整合音频信息显著提升了异常检测的准确性。特别值得注意的是，借助不确定度驱动的蒸馏增强单模态数据后，我们提出的方法始终优于现有的单模态视频异常检测方法。
URL
https://arxiv.org/abs/2504.04495
