Abstract
Recent advances in large language models (LLMs) have demonstrated strong performance on simple text classification tasks, frequently under zero-shot settings. However, their efficacy declines when tackling complex social media challenges such as propaganda detection, hateful meme classification, and toxicity identification. Much of the existing work has focused on using LLMs to generate synthetic training data, overlooking the potential of LLM-based text preprocessing and semantic augmentation. In this paper, we introduce an approach that prompts LLMs to clean noisy text and provide context-rich explanations, thereby enhancing training sets without substantial increases in data volume. We systematically evaluate on the SemEval 2024 multi-label Persuasive Meme dataset and further validate on the Google Jigsaw toxic comments and Facebook hateful memes datasets to assess generalizability. Our results reveal that zero-shot LLM classification underperforms on these high-context tasks compared to supervised models. In contrast, integrating LLM-based semantic augmentation yields performance on par with approaches that rely on human-annotated data, at a fraction of the cost. These findings underscore the importance of strategically incorporating LLMs into machine learning (ML) pipeline for social media classification tasks, offering broad implications for combating harmful content online.
Abstract (translated)
最近在大型语言模型（LLMs）方面取得的进展表明，这些模型在简单的文本分类任务上表现出色，并且通常可以在零样本设置下完成任务。然而，在处理复杂的社交媒体挑战如宣传检测、仇恨言论识别和毒性内容鉴定等情境时，它们的效果会显著下降。目前大部分研究都集中在使用LLM生成合成训练数据上，而忽略了基于LLM的文本预处理和语义增强的潜力。 在本文中，我们提出了一种新方法：通过提示LLM来清理嘈杂文本并提供丰富的背景解释，以此在不大幅增加数据量的情况下改进训练集。我们在SemEval 2024多标签说服性表情包（Persuasive Meme）数据集中进行了系统的评估，并进一步在谷歌Jigsaw有毒评论和Facebook仇恨言论表情包的数据集上进行验证以测试其泛化能力。我们的结果显示，零样本LLM分类在此类高上下文任务中表现不如监督模型。相比之下，结合基于LLM的语义增强方法，则可以达到与依赖人工标注数据的方法相媲美的性能，并且成本更低。 这些研究结果强调了在社交媒体分类任务中战略性的将大型语言模型集成到机器学习（ML）流程中的重要性，并为打击网上的有害内容提供了广泛的应用前景。
URL
https://arxiv.org/abs/2504.15548
