Abstract
Short-video misinformation detection has attracted wide attention in the multi-modal domain, aiming to accurately identify the misinformation in the video format accompanied by the corresponding audio. Despite significant advancements, current models in this field, trained on particular domains (source domains), often exhibit unsatisfactory performance on unseen domains (target domains) due to domain gaps. To effectively realize such domain generalization on the short-video misinformation detection task, we propose deep insights into the characteristics of different domains: (1) The detection on various domains may mainly rely on different modalities (i.e., mainly focusing on videos or audios). To enhance domain generalization, it is crucial to achieve optimal model performance on all modalities simultaneously. (2) For some domains focusing on cross-modal joint fraud, a comprehensive analysis relying on cross-modal fusion is necessary. However, domain biases located in each modality (especially in each frame of videos) will be accumulated in this fusion process, which may seriously damage the final identification of misinformation. To address these issues, we propose a new DOmain generalization model via ConsisTency and invariance learning for shORt-video misinformation detection (named DOCTOR), which contains two characteristic modules: (1) We involve the cross-modal feature interpolation to map multiple modalities into a shared space and the interpolation distillation to synchronize multi-modal learning; (2) We design the diffusion model to add noise to retain core features of multi modal and enhance domain invariant features through cross-modal guided denoising. Extensive experiments demonstrate the effectiveness of our proposed DOCTOR model. Our code is public available at this https URL.
Abstract (translated)
短视频误导信息检测在多模态领域引起了广泛关注，旨在准确识别伴随相应音频的视频格式中的虚假信息。尽管目前该领域的模型取得了显著进展，但这些模型通常仅针对特定领域（源域）进行训练，在未见过的领域（目标域）上表现不佳，原因是存在领域差距。为了在这种短视频误导信息检测任务中有效实现跨领域泛化，我们深入分析了不同领域的特点：(1) 不同领域的检测可能主要依赖于不同的模态（即主要关注视频或音频）。要增强跨领域泛化能力，关键在于同时在所有模态上达到最佳的模型性能。(2) 对于一些专注于跨模态联合欺诈的领域，需要进行基于跨模态融合的全面分析。然而，在这种融合过程中，各模态（特别是每个视频帧）中的领域偏差会被累积起来，这可能会严重损害最终的误导信息识别。 为解决这些问题，我们提出了一种新的跨域泛化模型——通过一致性和不变性学习实现短视频误导信息检测（名为DOCTOR），该模型包含两个特色模块：(1) 我们引入了跨模态特征插值来将多个模态映射到共享空间，并利用插值蒸馏同步多模态学习；(2) 我们设计了一种扩散模型，通过向多模态添加噪声保留核心特征并通过跨模态引导去噪增强领域不变性特征。广泛的实验展示了我们提出的DOCTOR模型的有效性。我们的代码可在以下链接获取：[此处插入实际的URL]。
URL
https://arxiv.org/abs/2507.04061
