Abstract
The rapid development of generative AI is a double-edged sword, which not only facilitates content creation but also makes image manipulation easier and more difficult to detect. Although current image forgery detection and localization (IFDL) methods are generally effective, they tend to face two challenges: \textbf{1)} black-box nature with unknown detection principle, \textbf{2)} limited generalization across diverse tampering methods (e.g., Photoshop, DeepFake, AIGC-Editing). To address these issues, we propose the explainable IFDL task and design FakeShield, a multi-modal framework capable of evaluating image authenticity, generating tampered region masks, and providing a judgment basis based on pixel-level and image-level tampering clues. Additionally, we leverage GPT-4o to enhance existing IFDL datasets, creating the Multi-Modal Tamper Description dataSet (MMTD-Set) for training FakeShield's tampering analysis capabilities. Meanwhile, we incorporate a Domain Tag-guided Explainable Forgery Detection Module (DTE-FDM) and a Multi-modal Forgery Localization Module (MFLM) to address various types of tamper detection interpretation and achieve forgery localization guided by detailed textual descriptions. Extensive experiments demonstrate that FakeShield effectively detects and localizes various tampering techniques, offering an explainable and superior solution compared to previous IFDL methods.
Abstract (translated)
生成式 AI 的快速发展是一把双刃剑，这不仅促进了内容创作，而且使图像编辑和检测变得更加容易和困难。尽管当前的图像伪造检测和局部定位（IFDL）方法通常有效，但它们往往面临两个挑战：（1）未知检测原理的黑盒性质，（2）不同编辑方法（如 Photoshop、DeepFake 和 AIGC 编辑）下的有限泛化能力。为解决这些问题，我们提出了可解释的 IFDL 任务，并设计了 FakeShield，一种多模态框架，能够评估图像真实性、生成修改区域mask，并提供基于像素级和图像级修改线索的判断依据。此外，我们还利用 GPT-4o 增强现有 IFDL 数据集，为训练 FakeShield 的修改分析能力创建了多模态 Tamper Description 数据集（MMTD-Set）。同时，我们引入了领域标签指导的伪造检测模块（DTE-FDM）和多模态伪造定位模块（MFLM），以解决各种修改检测解释和实现基于详细文本描述的伪造定位。大量实验证明，FakeShield 有效地检测和定位各种修改技术，与之前 IFDL 方法相比，提供了更高水平的有解释性和优越性。
URL
https://arxiv.org/abs/2410.02761
