Abstract
DeepFake based digital facial forgery is threatening public media security, especially when lip manipulation has been used in talking face generation, and the difficulty of fake video detection is further improved. By only changing lip shape to match the given speech, the facial features of identity are hard to be discriminated in such fake talking face videos. Together with the lack of attention on audio stream as the prior knowledge, the detection failure of fake talking face videos also becomes inevitable. It's found that the optical flow of the fake talking face video is disordered especially in the lip region while the optical flow of the real video changes regularly, which means the motion feature from optical flow is useful to capture manipulation cues. In this study, a fake talking face detection network (FTFDNet) is proposed by incorporating visual, audio and motion features using an efficient cross-modal fusion (CMF) module. Furthermore, a novel audio-visual attention mechanism (AVAM) is proposed to discover more informative features, which can be seamlessly integrated into any audio-visual CNN architecture by modularization. With the additional AVAM, the proposed FTFDNet is able to achieve a better detection performance than other state-of-the-art DeepFake video detection methods not only on the established fake talking face detection dataset (FTFDD) but also on the DeepFake video detection datasets (DFDC and DF-TIMIT).
Abstract (translated)
DeepFake based digital facial forgery威胁到了公共媒体安全,特别是在使用唇动技术生成对话面部视频时,而且假视频检测的难度进一步增加。仅改变唇形以匹配给定语音很难在这类假对话面部视频中找到身份特征。同时,缺乏对音频流作为先前知识的注意,也会导致假对话面部视频的检测失败。发现假对话面部视频的光学流特别不规律,而真实视频的光学流则变化规律,这意味着从光学流的动图特征可以用于捕捉操纵迹象。在本研究中,提出了一种假对话面部检测网络(FTFDNet),通过使用高效的跨模态融合模块(CMF)来集成视觉、音频和运动特征。此外,还提出了一种新的音频-视觉注意力机制(AVAM),以发现更多的有用特征,并通过模块化的方式将它们无缝集成到任何音频-视觉卷积神经网络架构中。额外的AVAM可以使提出的FTFDNet在建立的假对话面部检测数据集(FTFDD)和DeepFake视频检测数据集(DFDC和DF-TIMIT)上实现比现有先进的DeepFake视频检测方法更好的检测性能。
URL
https://arxiv.org/abs/2307.03990
