Abstract
Malicious image manipulation threatens public safety and requires efficient localization methods. Existing approaches depend on costly pixel-level annotations which make training expensive. Existing weakly supervised methods rely only on image-level binary labels and focus on global classification, often overlooking local edge cues that are critical for precise localization. We observe that feature variations at manipulated boundaries are substantially larger than in interior regions. To address this gap, we propose Semantic-Agnostic Prompt Learning (SAPL) in CLIP, which learns text prompts that intentionally encode non-semantic, boundary-centric cues so that CLIPs multimodal similarity highlights manipulation edges rather than high-level object semantics. SAPL combines two complementary modules Edge-aware Contextual Prompt Learning (ECPL) and Hierarchical Edge Contrastive Learning (HECL) to exploit edge information in both textual and visual spaces. The proposed ECPL leverages edge-enhanced image features to generate learnable textual prompts via an attention mechanism, embedding semantic-irrelevant information into text features, to guide CLIP focusing on manipulation edges. The proposed HECL extract genuine and manipulated edge patches, and utilize contrastive learning to boost the discrimination between genuine edge patches and manipulated edge patches. Finally, we predict the manipulated regions from the similarity map after processing. Extensive experiments on multiple public benchmarks demonstrate that SAPL significantly outperforms existing approaches, achieving state-of-the-art localization performance.
Abstract (translated)
恶意图像篡改威胁到公共安全，需要高效的定位方法。现有方法依赖于昂贵的像素级标注，使得训练成本高昂。现有的弱监督方法仅依靠图级别二进制标签，并专注于全局分类，往往忽视了局部边缘线索对于精确定位至关重要的作用。我们观察到，在被篡改边界上的特征变化远大于内部区域的变化。为解决这一问题，我们在CLIP模型中提出了语义不可知提示学习（SAPL），该方法通过有意编码非语义的边界中心线索来学习文本提示，使CLIP的多模态相似性更能突出篡改边缘而不是高层次的对象语义。SAPL结合了两个互补模块：边缘感知上下文提示学习（ECPL）和分层边缘对比学习（HECL），用于在文本空间和视觉空间中利用边缘信息。 所提出的ECPL通过增强图像特征的边缘，生成可学习的文字提示，并借助注意力机制将语义无关的信息嵌入到文字特征中，引导CLIP关注篡改边缘。而所提出的HECL则提取真实边界片和被篡改边界片，并利用对比学习来提高对真实边界片与被篡改边界片之间的区分能力。 最后，在处理完相似性地图后，我们预测出被篡改的区域。在多个公开基准上的广泛实验表明，SAPL显著优于现有方法，实现了最先进的定位性能。
URL
https://arxiv.org/abs/2601.06222
