Abstract
The spread of Deepfake videos has caused a trust crisis and impaired social stability. Although numerous approaches have been proposed to address the challenges of Deepfake detection and localization, there is still a lack of systematic research on the weakly supervised multimodal fine-grained temporal forgery localization (WS-MTFL). In this paper, we propose a novel weakly supervised multimodal temporal forgery localization via multitask learning (WMMT), which addresses the WS-MTFL under the multitask learning paradigm. WMMT achieves multimodal fine-grained Deepfake detection and temporal partial forgery localization using merely video-level annotations. Specifically, visual and audio modality detection are formulated as two binary classification tasks. The multitask learning paradigm is introduced to integrate these tasks into a multimodal task. Furthermore, WMMT utilizes a Mixture-of-Experts structure to adaptively select appropriate features and localization head, achieving excellent flexibility and localization precision in WS-MTFL. A feature enhancement module with temporal property preserving attention mechanism is proposed to identify the intra- and inter-modality feature deviation and construct comprehensive video features. To further explore the temporal information for weakly supervised learning, an extensible deviation perceiving loss has been proposed, which aims to enlarge the deviation of adjacent segments of the forged samples and reduce the deviation of genuine samples. Extensive experiments demonstrate the effectiveness of multitask learning for WS-MTFL, and the WMMT achieves comparable results to fully supervised approaches in several evaluation metrics.
Abstract (translated)
深度伪造视频的传播已经导致了信任危机，并影响到了社会稳定性。尽管已提出了许多方法来解决深度伪造检测和定位所面临的挑战，但弱监督下的多模态精细时间伪造定位（WS-MTFL）的研究仍存在系统性不足的问题。本文提出了一种基于多任务学习的新颖弱监督多模态时序伪造定位方法（WMMT），该方法在多任务学习范式下解决了WS-MTFL问题。WMMT仅通过视频级标注就能实现多模态精细深度伪造检测和时间局部伪造的定位。具体来说，视觉和音频模式检测被构造成两个二分类任务。引入了多任务学习框架将这些任务整合成一个多模态任务。此外，WMMT采用了混合专家（Mixture-of-Experts）结构，能够自适应地选择合适的特征及定位头，在WS-MTFL中实现了出色的灵活性和定位精度。为了识别出同一模式内的特征偏差以及跨模式间的特征差异，并构建全面的视频特征，本文提出了一种具有时间特性保持注意机制的功能增强模块。为进一步探索弱监督学习中的时序信息，引入了一种可扩展的偏差感知损失函数，其目标是扩大伪造样本相邻段落之间的偏差并减小真实样本的偏差。 大量的实验表明，多任务学习对于WS-MTFL的有效性，并且WMMT在多个评价指标中实现了与完全监督方法相当的结果。
URL
https://arxiv.org/abs/2508.02179
