Abstract
Nowadays, the widespread dissemination of misinformation across numerous social media platforms has led to severe negative effects on society. To address this challenge, the automatic detection of misinformation, particularly under multimedia scenarios, has gained significant attention from both academic and industrial communities, leading to the emergence of a research task known as Multimodal Misinformation Detection (MMD). Typically, current MMD approaches focus on capturing the semantic relationships and inconsistency between various modalities but often overlook certain critical indicators within multimodal content. Recent research has shown that manipulated features within visual content in social media articles serve as valuable clues for MMD. Meanwhile, we argue that the potential intentions behind the manipulation, e.g., harmful and harmless, also matter in MMD. Therefore, in this study, we aim to identify such multimodal misinformation by capturing two types of features: manipulation features, which represent if visual content has been manipulated, and intention features, which assess the nature of these manipulations, distinguishing between harmful and harmless intentions. Unfortunately, the manipulation and intention labels that supervise these features to be discriminative are unknown. To address this, we introduce two weakly supervised indicators as substitutes by incorporating supplementary datasets focused on image manipulation detection and framing two different classification tasks as positive and unlabeled learning issues. With this framework, we introduce an innovative MMD approach, titled Harmful Visual Content Manipulation Matters in MMD (HAVC-M4 D). Comprehensive experiments conducted on four prevalent MMD datasets indicate that HAVC-M4 D significantly and consistently enhances the performance of existing MMD methods.
Abstract (translated)
如今，虚假信息在众多社交媒体平台上的广泛传播已对社会造成严重负面影响。为应对这一挑战，尤其在多媒体场景下对虚假信息的自动检测已备受学术界与工业界关注，并催生了名为多模态虚假信息检测（Multimodal Misinformation Detection, MMD）的研究任务。通常，当前MMD方法侧重于捕捉多模态间的语义关联与不一致性，却常忽略多模态内容中的某些关键指标。近期研究表明，社交媒体文章中视觉内容的篡改特征可作为MMD的重要线索。同时，我们认为篡改行为背后的潜在意图（如有害与无害）同样对MMD至关重要。因此，本研究旨在通过捕捉两类特征识别此类多模态虚假信息：一是表征视觉内容是否被篡改的篡改特征，二是评估篡改性质、区分有害与无害意图的意图特征。然而，监督这些特征具有判别性的篡改标签与意图标签均未知。为解决此问题，我们引入两种弱监督指标作为替代：通过整合专注于图像篡改检测的辅助数据集，并将两种不同分类任务构建为正与未标注学习问题。基于此框架，我们提出一种创新性MMD方法，命名为《MMD中有害视觉内容篡改至关重要》（Harmful Visual Content Manipulation Matters in MMD, HAVC-M4 D）。在四个主流MMD数据集上开展的全面实验表明，HAVC-M4 D能显著且稳定地提升现有MMD方法的性能。
URL
https://arxiv.org/abs/2603.21054
