Abstract
Amid a tidal wave of misinformation flooding social media during elections and crises, extensive research has been conducted on misinformation detection, primarily focusing on text-based or image-based approaches. However, only a few studies have explored multimodal feature combinations, such as integrating text and images for building a classification model to detect misinformation. This study investigates the effectiveness of different multimodal feature combinations, incorporating text, images, and social features using an early fusion approach for the classification model. This study analyzed 1,529 tweets containing both text and images during the COVID-19 pandemic and election periods collected from Twitter (now X). A data enrichment process was applied to extract additional social features, as well as visual features, through techniques such as object detection and optical character recognition (OCR). The results show that combining unsupervised and supervised machine learning models improves classification performance by 15% compared to unimodal models and by 5% compared to bimodal models. Additionally, the study analyzes the propagation patterns of misinformation based on the characteristics of misinformation tweets and the users who disseminate them.
Abstract (translated)
在选举和危机期间，随着社交媒体上错误信息泛滥成灾，已经进行了大量关于错误信息检测的研究，主要集中在基于文本或图像的方法。然而，只有少数研究探讨了多模态特征组合的使用，例如将文本与图像相结合以构建用于检测错误信息的分类模型。这项研究调查了不同多模态特征组合的有效性，通过早期融合方法在分类模型中结合文本、图片和社会特征进行分析。该研究分析了1,529条包含文本和图片的推特，在新冠疫情期间以及选举期间从Twitter（现为X）收集的数据。 为了提取额外的社会特征及视觉特征，采用数据增强过程，并应用如对象检测和光学字符识别（OCR）等技术。结果显示，结合无监督和有监督机器学习模型能比单一模态模型提高15%的分类性能，相较于双模态模型也有5%的提升。此外，该研究还根据错误信息推文的特征及其传播者的特点分析了错误信息的传播模式。
URL
https://arxiv.org/abs/2507.01984
