Text_Classification
-
Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text
2025-02-18 15:36:16Andrei Jarca, Florinel Alin Croitoru, Radu Tudor IonescuAbstract
Masked language modeling has become a widely adopted unsupervised technique to pre-train language models. However, the process of selecting tokens for masking is random, and the percentage of masked tokens is typically fixed for the entire training process. In this paper, we propose to adjust the masking ratio and to decide which tokens to mask based on a novel task-informed anti-curriculum learning scheme. First, we harness task-specific knowledge about useful and harmful tokens in order to determine which tokens to mask. Second, we propose a cyclic decaying masking ratio, which corresponds to an anti-curriculum schedule (from hard to easy). We exemplify our novel task-informed anti-curriculum by masking (TIACBM) approach across three diverse downstream tasks: sentiment analysis, text classification by topic, and authorship attribution. Our findings suggest that TIACBM enhances the ability of the model to focus on key task-relevant features, contributing to statistically significant performance gains across tasks. We release our code at this https URL.
Abstract (translated)
遮罩语言模型已成为预训练语言模型中广泛采用的无监督技术。然而，选择进行遮蔽的标记的过程是随机的，并且在整个训练过程中被遮蔽标记的比例通常是固定的。在本文中，我们提出调整遮蔽比例并根据一种新的任务相关反课程学习方案来决定要遮蔽哪些标记。首先，我们利用特定于任务的知识来确定哪些标记是有用的或有害的，从而决定要遮蔽哪些标记。其次，我们提出了一个周期性衰减的遮蔽比率，这对应于从难到易的反课程安排。 我们通过三种不同的下游任务展示了我们的新型任务相关反课程方法：情感分析、按主题分类和作者归属。实验结果表明，TIACBM（Task-Informed Anti-Curriculum Based Masking）能够增强模型聚焦于关键的任务相关信息的能力，在各个任务上带来了统计显著的性能提升。我们在提供的链接中发布了我们的代码。 原文链接： [此处应插入实际的URL](this https URL)
URL
https://arxiv.org/abs/2502.12953
PDF
-
AdaSplash: Adaptive Sparse Flash Attention
2025-02-17 17:56:23Nuno Gon\c{c}alves, Marcos Treviso, Andr\'e F. T. MartinsAbstract
The computational cost of softmax-based attention in transformers limits their applicability to long-context tasks. Adaptive sparsity, of which $\alpha$-entmax attention is an example, offers a flexible data-dependent alternative, but existing implementations are inefficient and do not leverage the sparsity to obtain runtime and memory gains. In this work, we propose AdaSplash, which combines the efficiency of GPU-optimized algorithms with the sparsity benefits of $\alpha$-entmax. We first introduce a hybrid Halley-bisection algorithm, resulting in a 7-fold reduction in the number of iterations needed to compute the $\alpha$-entmax transformation. Then, we implement custom Triton kernels to efficiently handle adaptive sparsity. Experiments with RoBERTa and ModernBERT for text classification and single-vector retrieval, along with GPT-2 for language modeling, show that our method achieves substantial improvements in runtime and memory efficiency compared to existing $\alpha$-entmax implementations. It approaches -- and in some cases surpasses -- the efficiency of highly optimized softmax implementations like FlashAttention-2, enabling long-context training while maintaining strong task performance.
Abstract (translated)
基于softmax的注意力机制在变压器中的计算成本限制了其在长上下文任务中的适用性。自适应稀疏性，例如$\alpha$-entmax注意就是此类方法的一个例子，提供了一种灵活的数据依赖性替代方案，但现有的实现效率低下，并没有利用稀疏性来获得运行时间和内存的节省。在这项工作中，我们提出了AdaSplash，它结合了GPU优化算法的效率与$\alpha$-entmax的稀疏性优势。首先，我们引入了一个混合Halley-二分法算法，使得计算$\alpha$-entmax变换所需的迭代次数减少了7倍。然后，我们通过自定义Triton内核有效地处理适应性稀疏性问题。使用RoBERTa和ModernBERT进行文本分类和单向量检索的实验，以及GPT-2用于语言建模，证明了我们的方法在运行时间和内存效率方面相比现有的$\alpha$-entmax实现取得了显著改进。我们的方法接近于——并在某些情况下超过了——高度优化的softmax实现（如FlashAttention-2）的效率，这使得能够进行长上下文训练的同时保持强大的任务性能。
URL
https://arxiv.org/abs/2502.12082
PDF
-
Text Classification in the LLM Era - Where do we stand?
2025-02-17 14:25:54Sowmya Vajjala, Shwetali ShimangaudAbstract
Large Language Models revolutionized NLP and showed dramatic performance improvements across several tasks. In this paper, we investigated the role of such language models in text classification and how they compare with other approaches relying on smaller pre-trained language models. Considering 32 datasets spanning 8 languages, we compared zero-shot classification, few-shot fine-tuning and synthetic data based classifiers with classifiers built using the complete human labeled dataset. Our results show that zero-shot approaches do well for sentiment classification, but are outperformed by other approaches for the rest of the tasks, and synthetic data sourced from multiple LLMs can build better classifiers than zero-shot open LLMs. We also see wide performance disparities across languages in all the classification scenarios. We expect that these findings would guide practitioners working on developing text classification systems across languages.
Abstract (translated)
大型语言模型在自然语言处理（NLP）领域引发了革命，展现了多项任务上的显著性能提升。在这篇论文中，我们研究了此类语言模型在文本分类中的作用，并将其与依赖于较小预训练语言模型的其他方法进行了比较。考虑到涵盖8种语言的32个数据集，我们将零样本分类、少量样本微调和基于合成数据的分类器与使用完整人工标注数据集构建的分类器进行对比。我们的结果显示，对于情感分类，零样本方法表现良好；但对于其余任务，则被其他方法超越。此外，来自多个大型语言模型（LLM）的合成数据可以建立比零样本开放语言模型更好的分类器。在所有文本分类场景中，我们还观察到了跨不同语言间的广泛性能差异。我们认为这些发现将有助于指导在多语言环境下开发文本分类系统的研究人员和实践者。
URL
https://arxiv.org/abs/2502.11830
PDF
-
UNITE-FND: Reframing Multimodal Fake News Detection through Unimodal Scene Translation
2025-02-16 14:00:57Arka Mukherjee, Shreya GhoshAbstract
Multimodal fake news detection typically demands complex architectures and substantial computational resources, posing deployment challenges in real-world settings. We introduce UNITE-FND, a novel framework that reframes multimodal fake news detection as a unimodal text classification task. We propose six specialized prompting strategies with Gemini 1.5 Pro, converting visual content into structured textual descriptions, and enabling efficient text-only models to preserve critical visual information. To benchmark our approach, we introduce Uni-Fakeddit-55k, a curated dataset family of 55,000 samples each, each processed through our multimodal-to-unimodal translation framework. Experimental results demonstrate that UNITE-FND achieves 92.52% accuracy in binary classification, surpassing prior multimodal models while reducing computational costs by over 10x (TinyBERT variant: 14.5M parameters vs. 250M+ in SOTA models). Additionally, we propose a comprehensive suite of five novel metrics to evaluate image-to-text conversion quality, ensuring optimal information preservation. Our results demonstrate that structured text-based representations can replace direct multimodal processing with minimal loss of accuracy, making UNITE-FND a practical and scalable alternative for resource-constrained environments.
Abstract (translated)
多模态假新闻检测通常需要复杂的架构和大量的计算资源，这在实际部署中带来了挑战。我们提出了UNITE-FND框架，该框架将多模态假新闻检测重新定义为单模态文本分类任务。我们与Gemini 1.5 Pro合作，提出六种专门的提示策略，能够将视觉内容转化为结构化的文本描述，并使纯文本模型能够高效地保留关键的视觉信息。为了评估我们的方法，我们引入了Uni-Fakeddit-55k数据集家族，包含经过我们多模态到单模态转换框架处理后的55,000个样本。实验结果显示，UNITE-FND在二元分类中达到了92.52%的准确率，超过了之前的多模态模型，并将计算成本减少了超过10倍（TinyBERT变体：14.5M参数 vs 最先进的模型中的2.5亿+）。此外，我们还提出了一套五种新的评估指标来衡量图像到文本转换的质量，以确保最佳的信息保留。我们的研究结果表明，基于结构化文本的表示可以替代直接的多模态处理，并且在准确性上几乎没有损失，这使得UNITE-FND成为资源受限环境中的实用和可扩展的解决方案。
URL
https://arxiv.org/abs/2502.11132
PDF
-
Towards Achieving Concept Completeness for Unsupervised Textual Concept Bottleneck Models
2025-02-16 12:28:43Milan Bhan, Yann Choho, Pierre Moreau, Jean-Noel Vittaut, Nicolas Chesneau, Marie-Jeanne LesotAbstract
Textual Concept Bottleneck Models (TBMs) are interpretable-by-design models for text classification that predict a set of salient concepts before making the final prediction. This paper proposes Complete Textual Concept Bottleneck Model (CT-CBM),a novel TCBM generator building concept labels in a fully unsupervised manner using a small language model, eliminating both the need for predefined human labeled concepts and LLM annotations. CT-CBM iteratively targets and adds important concepts in the bottleneck layer to create a complete concept basis and addresses downstream classification leakage through a parallel residual connection. CT-CBM achieves good results against competitors, offering a promising solution to enhance interpretability of NLP classifiers without sacrificing performance.
Abstract (translated)
文本概念瓶颈模型（TBMs）是一种设计上可解释的文本分类模型，它们在做出最终预测之前会先预测一组显著的概念。本文提出了一种全新的TBMs生成器——Complete Textual Concept Bottleneck Model (CT-CBM)，该模型完全采用无监督的方式使用小型语言模型来构建概念标签，从而消除了对预定义的人工标注概念和大型语言模型注释的需求。CT-CBM通过在瓶颈层迭代地添加重要的概念来创建一个完整的概念基础，并且通过并行残差连接解决了下游分类泄漏的问题。实验结果表明，CT-CBM在与竞争对手的比较中表现良好，为提高NLP分类器的可解释性而不牺牲性能提供了有希望的解决方案。
URL
https://arxiv.org/abs/2502.11100
PDF
-
Retrieval-augmented Encoders for Extreme Multi-label Text Classification
2025-02-15 00:30:28Yau-Shian Wang, Wei-Cheng Chang, Jyun-Yu Jiang, Jiong Zhang, Hsiang-Fu Yu, S. V. N. VishwanathanAbstract
Extreme multi-label classification (XMC) seeks to find relevant labels from an extremely large label collection for a given text input. To tackle such a vast label space, current state-of-the-art methods fall into two categories. The one-versus-all (OVA) method uses learnable label embeddings for each label, excelling at memorization (i.e., capturing detailed training signals for accurate head label prediction). In contrast, the dual-encoder (DE) model maps input and label text into a shared embedding space for better generalization (i.e., the capability of predicting tail labels with limited training data), but may fall short at memorization. To achieve generalization and memorization, existing XMC methods often combine DE and OVA models, which involves complex training pipelines. Inspired by the success of retrieval-augmented language models, we propose the Retrieval-augmented Encoders for XMC (RAEXMC), a novel framework that equips a DE model with retrieval-augmented capability for efficient memorization without additional trainable parameter. During training, RAEXMC is optimized by the contrastive loss over a knowledge memory that consists of both input instances and labels. During inference, given a test input, RAEXMC retrieves the top-$K$ keys from the knowledge memory, and aggregates the corresponding values as the prediction scores. We showcase the effectiveness and efficiency of RAEXMC on four public LF-XMC benchmarks. RAEXMC not only advances the state-of-the-art (SOTA) DE method DEXML, but also achieves more than 10x speedup on the largest LF-AmazonTitles-1.3M dataset under the same 8 A100 GPUs training environments.
Abstract (translated)
极端多标签分类（XMC）的目标是从一个极其庞大的标签集合中找出与给定文本输入相关的标签。为了应对这种大规模的标签空间，当前最先进的方法主要分为两大类：一种是使用可学习的标签嵌入的“一对所有”（OVA）方法，这种方法在记忆化方面表现出色，即能够捕捉到详细的训练信号以准确预测头部标签；另一种是双编码器（DE）模型，该模型将输入文本和标签映射到一个共享的嵌入空间中，从而更好地泛化（即，在有限的数据下也能有效预测尾部标签），但可能在记忆化方面略有不足。现有的XMC方法通常通过结合使用DE和OVA模型来实现泛化和记忆化的效果，这涉及复杂的训练流程。 受检索增强语言模型成功的启发，我们提出了用于XMC的检索增强编码器（RAEXMC）框架。该框架为双编码器模型配备了检索增强能力，以高效地进行记忆化，而不增加额外可学习参数的数量。在训练阶段，通过对比损失优化RAEXMC，并利用一个包含输入实例和标签的知识内存来实现这一目的。在推理阶段，给定一个测试输入后，RAEXMC从知识内存中检索出前K个键，然后将对应的值聚合为预测得分。 我们在四个公开的LF-XMC基准数据集上展示了RAEXMC的有效性和效率。不仅比现有的最先进的双编码器方法DEXML有所改进，而且在最大的LF-AmazonTitles-1.3M数据集中，在相同条件下使用8块A100 GPU进行训练时，速度提高了超过10倍。
URL
https://arxiv.org/abs/2502.10615
PDF
-
A Hybrid Model for Few-Shot Text Classification Using Transfer and Meta-Learning
2025-02-13 09:00:32Jia Gao, Shuangquan Lyu, Guiran Liu, Binrong Zhu, Hongye Zheng, Xiaoxuan LiaoAbstract
With the continuous development of natural language processing (NLP) technology, text classification tasks have been widely used in multiple application fields. However, obtaining labeled data is often expensive and difficult, especially in few-shot learning scenarios. To solve this problem, this paper proposes a few-shot text classification model based on transfer learning and meta-learning. The model uses the knowledge of the pre-trained model for transfer and optimizes the model's rapid adaptability in few-sample tasks through a meta-learning mechanism. Through a series of comparative experiments and ablation experiments, we verified the effectiveness of the proposed method. The experimental results show that under the conditions of few samples and medium samples, the model based on transfer learning and meta-learning significantly outperforms traditional machine learning and deep learning methods. In addition, ablation experiments further analyzed the contribution of each component to the model performance and confirmed the key role of transfer learning and meta-learning in improving model accuracy. Finally, this paper discusses future research directions and looks forward to the potential of this method in practical applications.
Abstract (translated)
随着自然语言处理（NLP）技术的不断进步，文本分类任务已经在多个应用领域得到了广泛的应用。然而，在获取标注数据时往往成本高昂且难度较大，特别是在小样本学习场景中尤为明显。为解决这一问题，本文提出了一种基于迁移学习和元学习的少量样本文本分类模型。该模型利用预训练模型的知识进行知识转移，并通过元学习机制优化模型在小样本任务中的快速适应能力。通过一系列对比实验和消融实验，我们验证了所提方法的有效性。实验结果显示，在小样本和中等样本条件下，基于迁移学习和元学习的模型明显优于传统的机器学习和深度学习方法。此外，消融实验证明了每个组件对模型性能的贡献，并确认了迁移学习与元学习在提高模型准确性中的关键作用。最后，本文探讨了未来的研究方向，并展望了该方法在实际应用中的潜力。
URL
https://arxiv.org/abs/2502.09086
PDF
-
Measuring Diversity in Synthetic Datasets
2025-02-12 15:46:34Yuchang Zhu, Huizhe Zhang, Bingzhe Wu, Jintang Li, Zibin Zheng, Peilin Zhao, Liang Chen, Yatao BianAbstract
Large language models (LLMs) are widely adopted to generate synthetic datasets for various natural language processing (NLP) tasks, such as text classification and summarization. However, accurately measuring the diversity of these synthetic datasets-an aspect crucial for robust model performance-remains a significant challenge. In this paper, we introduce DCScore, a novel method for measuring synthetic dataset diversity from a classification perspective. Specifically, DCScore formulates diversity evaluation as a sample classification task, leveraging mutual relationships among samples. We further provide theoretical verification of the diversity-related axioms satisfied by DCScore, highlighting its role as a principled diversity evaluation method. Experimental results on synthetic datasets reveal that DCScore enjoys a stronger correlation with multiple diversity pseudo-truths of evaluated datasets, underscoring its effectiveness. Moreover, both empirical and theoretical evidence demonstrate that DCScore substantially reduces computational costs compared to existing approaches. Code is available at: this https URL.
Abstract (translated)
大型语言模型（LLMs）被广泛应用于生成各种自然语言处理（NLP）任务的合成数据集，例如文本分类和摘要。然而，准确衡量这些合成数据集的多样性——这对于实现稳健的模型性能至关重要——仍然是一个重大挑战。在本文中，我们介绍了DCScore，这是一种从分类角度测量合成数据集多样性的新方法。具体来说，DCScore将多样性评估表述为一个样本分类任务，并利用了样本之间的相互关系。此外，我们还提供了关于DCScore所满足的多样性相关公理的理论验证，强调其作为原则性多样性评价方法的角色。在合成数据集上的实验结果表明，DCScore与被评估的数据集中多个多样性的伪真实情况之间存在更强的相关性，这凸显了它的有效性。而且，无论是经验还是理论证据都证明，相较于现有方法，DCScore能够显著降低计算成本。代码可在以下链接获取：[此URL]（请将“this https URL”替换为实际提供的GitHub或其它版本控制系统的链接）。
URL
https://arxiv.org/abs/2502.08512
PDF
-
Don't Just Demo, Teach Me the Principles: A Principle-Based Multi-Agent Prompting Strategy for Text Classification
2025-02-11 01:10:13Peipei Wei, Dimitris Dimitriadis, Yan Xu, Mingwei ShenAbstract
We present PRINCIPLE-BASED PROMPTING, a simple but effective multi-agent prompting strategy for text classification. It first asks multiple LLM agents to independently generate candidate principles based on analysis of demonstration samples with or without labels, consolidates them into final principles via a finalizer agent, and then sends them to a classifier agent to perform downstream classification tasks. Extensive experiments on binary and multi-class classification datasets with different sizes of LLMs show that our approach not only achieves substantial performance gains (1.55% - 19.37%) over zero-shot prompting on macro-F1 score but also outperforms other strong baselines (CoT and stepback prompting). Principles generated by our approach help LLMs perform better on classification tasks than human crafted principles on two private datasets. Our multi-agent PRINCIPLE-BASED PROMPTING approach also shows on-par or better performance compared to demonstration-based few-shot prompting approaches, yet with substantially lower inference costs. Ablation studies show that label information and the multi-agent cooperative LLM framework play an important role in generating high-quality principles to facilitate downstream classification tasks.
Abstract (translated)
我们提出了一种基于原则的提示策略（PRINCIPLE-BASED PROMPTING），这是一种简单但有效的多代理文本分类提示方法。首先，该方法让多个大型语言模型（LLM）代理独立生成候选原则，这些候选原则是通过对带有或不带标签的示范样本进行分析得出的。随后，通过一个最终化器代理将这些候选原则整合成最终的原则，并将其发送给分类器代理以执行下游分类任务。 在不同大小的语言模型上的二元和多类分类数据集上进行了广泛的实验表明，我们的方法不仅在宏平均F1得分上比零样本提示获得了显著的性能提升（1.55% - 19.37%），而且也超过了其他强大的基准（如CoT 和 stepback 提示）。由我们方法生成的原则帮助语言模型在两个私有数据集上的分类任务中优于人工创建的原则。此外，我们的多代理 PRINCIPLE-BASED PROMPTING 方法显示出了与基于示范的少量样本提示方法相当或更好的性能表现，但推理成本却大大降低。 消融研究还表明，标签信息和多代理协同语言模型框架在生成高质量原则以促进下游分类任务方面发挥了重要作用。
URL
https://arxiv.org/abs/2502.07165
PDF
-
Efficient Scientific Full Text Classification: The Case of EICAT Impact Assessments
2025-02-10 15:19:22Marc Felix Brinner, Sina Zarrie{\ss}Abstract
This study explores strategies for efficiently classifying scientific full texts using both small, BERT-based models and local large language models like Llama-3.1 8B. We focus on developing methods for selecting subsets of input sentences to reduce input size while simultaneously enhancing classification performance. To this end, we compile a novel dataset consisting of full-text scientific papers from the field of invasion biology, specifically addressing the impacts of invasive species. These papers are aligned with publicly available impact assessments created by researchers for the International Union for Conservation of Nature (IUCN). Through extensive experimentation, we demonstrate that various sources like human evidence annotations, LLM-generated annotations or explainability scores can be used to train sentence selection models that improve the performance of both encoder- and decoder-based language models while optimizing efficiency through the reduction in input length, leading to improved results even if compared to models like ModernBERT that are able to handle the complete text as input. Additionally, we find that repeated sampling of shorter inputs proves to be a very effective strategy that, at a slightly increased cost, can further improve classification performance.
Abstract (translated)
这项研究探讨了利用基于BERT的小型模型和本地大型语言模型（如Llama-3.1 8B）高效分类科学全文的方法。我们专注于开发选择输入句子子集的策略，以减少输入大小同时提升分类性能。为此，我们编译了一个新颖的数据集，其中包括来自入侵生物学领域的完整文本科学论文，特别关注外来物种的影响。这些论文与国际自然保护联盟（IUCN）的研究人员为公众提供的影响评估相一致。 通过广泛的实验，我们展示了人类证据注释、大型语言模型生成的注释或可解释性评分等各种来源可以用于训练句子选择模型，从而在减少输入长度的同时提高基于编码器和解码器的语言模型的性能。即使与能够处理完整文本输入的现代BERT等模型相比，这种方法也能带来更好的结果。此外，我们发现重复采样较短的输入是一种非常有效的策略，在略微增加成本的情况下可以进一步提升分类性能。
URL
https://arxiv.org/abs/2502.06551
PDF
-
Benchmarking Prompt Sensitivity in Large Language Models
2025-02-09 23:01:03Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, Ebrahim BagheriAbstract
Large language Models (LLMs) are highly sensitive to variations in prompt formulation, which can significantly impact their ability to generate accurate responses. In this paper, we introduce a new task, Prompt Sensitivity Prediction, and a dataset PromptSET designed to investigate the effects of slight prompt variations on LLM performance. Using TriviaQA and HotpotQA datasets as the foundation of our work, we generate prompt variations and evaluate their effectiveness across multiple LLMs. We benchmark the prompt sensitivity prediction task employing state-of-the-art methods from related tasks, including LLM-based self-evaluation, text classification, and query performance prediction techniques. Our findings reveal that existing methods struggle to effectively address prompt sensitivity prediction, underscoring the need to understand how information needs should be phrased for accurate LLM responses.
Abstract (translated)
大型语言模型（LLMs）对提示语的细微变化非常敏感，这会显著影响它们生成准确回复的能力。本文介绍了新的任务——提示语敏感性预测（Prompt Sensitivity Prediction），以及一个名为PromptSET的数据集，用于研究轻微的提示语变化如何影响LLM的表现。基于TriviaQA和HotpotQA数据集，我们生成了不同的提示语变体，并对多个大型语言模型进行了有效性评估。我们使用来自相关任务的前沿方法来基准测试提示敏感性预测任务，包括基于LLM的自我评价、文本分类以及查询性能预测技术等。研究发现表明，现有的方法难以有效解决提示敏感性预测问题，这凸显了理解信息需求如何表述以获得准确的LLM回复的重要性。
URL
https://arxiv.org/abs/2502.06065
PDF
-
ARISE: Iterative Rule Induction and Synthetic Data Generation for Text Classification
2025-02-09 14:39:01Yashwanth M., Vaibhav Singh, Ayush Maheshwari, Amrith Krishna, Ganesh RamakrishnanAbstract
We propose ARISE, a framework that iteratively induces rules and generates synthetic data for text classification. We combine synthetic data generation and automatic rule induction, via bootstrapping, to iteratively filter the generated rules and data. We induce rules via inductive generalisation of syntactic n-grams, enabling us to capture a complementary source of supervision. These rules alone lead to performance gains in both, in-context learning (ICL) and fine-tuning (FT) settings. Similarly, use of augmented data from ARISE alone improves the performance for a model, outperforming configurations that rely on complex methods like contrastive learning. Further, our extensive experiments on various datasets covering three full-shot, eight few-shot and seven multilingual variant settings demonstrate that the rules and data we generate lead to performance improvements across these diverse domains and languages.
Abstract (translated)
我们提出了ARISE框架，该框架通过迭代地诱导规则并生成合成数据来实现文本分类。我们结合了合成数据生成和自动规则归纳，并通过引导式方法迭代地过滤生成的规则和数据。我们利用语法n-gram的归纳概括来诱导规则，这使我们能够捕捉到一种补充性的监督来源。仅凭这些规则，在上下文学习（ICL）和微调（FT）设置中都能实现性能提升。同样，单独使用ARISE产生的增强数据也能提高模型的性能，并且优于依赖复杂方法如对比学习等配置。 此外，我们在涵盖三个全样本、八个少量样本以及七个多语言变体设置的各种数据集上进行了广泛的实验，结果表明我们生成的规则和数据能够跨这些多样化的领域和语言实现性能提升。
URL
https://arxiv.org/abs/2502.05923
PDF
-
Enhancing Disinformation Detection with Explainable AI and Named Entity Replacement
2025-02-07 12:01:26Santiago Gonz\'alez-Silot, Andr\'es Montoro-Montarroso, Eugenio Mart\'inez C\'amara, Juan G\'omez-RomeroAbstract
The automatic detection of disinformation presents a significant challenge in the field of natural language processing. This task addresses a multifaceted societal and communication issue, which needs approaches that extend beyond the identification of general linguistic patterns through data-driven algorithms. In this research work, we hypothesise that text classification methods are not able to capture the nuances of disinformation and they often ground their decision in superfluous features. Hence, we apply a post-hoc explainability method (SHAP, SHapley Additive exPlanations) to identify spurious elements with high impact on the classification models. Our findings show that non-informative elements (e.g., URLs and emoticons) should be removed and named entities (e.g., Rwanda) should be pseudo-anonymized before training to avoid models' bias and increase their generalization capabilities. We evaluate this methodology with internal dataset and external dataset before and after applying extended data preprocessing and named entity replacement. The results show that our proposal enhances on average the performance of a disinformation classification method with external test data in 65.78% without a significant decrease of the internal test performance.
Abstract (translated)
自动检测虚假信息在自然语言处理领域中面临重大挑战。这一任务涉及到多方面的社会和沟通问题，需要超越通过数据驱动算法识别一般语言模式的方法。在这项研究工作中，我们假设文本分类方法无法捕捉到虚假信息的细微差别，并且它们通常依赖于无关紧要的特征进行决策。因此，我们应用了一种事后可解释性方法（SHAP, SHapley Additive exPlanations）来识别对分类模型影响较大的不相关元素。我们的研究发现表明，在训练之前应该移除非相关信息（例如网址和表情符号），并匿名化命名实体（例如卢旺达）以避免模型偏差，提高其泛化能力。 我们使用内部数据集和外部数据集在应用扩展的数据预处理和命名实体替换前后评估了这一方法。结果显示，我们的提议方法平均提高了65.78%的虚假信息分类方法在外部测试数据上的性能，并且没有显著降低内部测试的表现。
URL
https://arxiv.org/abs/2502.04863
PDF
-
How does a Multilingual LM Handle Multiple Languages?
2025-02-06 18:08:14Santhosh Kakarla, Gautama Shastry Bulusu Venkata, Aishwarya GaddamAbstract
Multilingual language models have significantly advanced due to rapid progress in natural language processing. Models like BLOOM 1.7B, trained on diverse multilingual datasets, aim to bridge linguistic gaps. However, their effectiveness in capturing linguistic knowledge, particularly for low-resource languages, remains an open question. This study critically examines MLMs capabilities in multilingual understanding, semantic representation, and cross-lingual knowledge transfer. While these models perform well for high-resource languages, they struggle with less-represented ones. Additionally, traditional evaluation methods often overlook their internal syntactic and semantic encoding. This research addresses key limitations through three objectives. First, it assesses semantic similarity by analyzing multilingual word embeddings for consistency using cosine similarity. Second, it examines BLOOM-1.7B and Qwen2 through Named Entity Recognition and sentence similarity tasks to understand their linguistic structures. Third, it explores cross-lingual knowledge transfer by evaluating generalization from high-resource to low-resource languages in sentiment analysis and text classification. By leveraging linguistic probing, performance metrics, and visualizations, this study provides insights into the strengths and limitations of MLMs. The findings aim to enhance multilingual NLP models, ensuring better support for both high- and low-resource languages, thereby promoting inclusivity in language technologies.
Abstract (translated)
多语言语言模型由于自然语言处理的迅速进步而得到了显著的发展。像BLOOM 1.7B这样的模型，在多样化的多语种数据集上进行训练，旨在弥合语言鸿沟。然而，它们在捕捉低资源语言的语言知识方面的有效性仍然是一个开放性问题。这项研究从批判的角度审视了多语言语言模型（MLMs）在跨语言理解、语义表示和跨语言知识迁移方面的能力。虽然这些模型对于高资源语言表现良好，但在处理代表性不足的语言时却面临挑战。此外，传统的评估方法通常忽视了它们内部的句法和语义编码。这项研究通过三个目标来解决关键限制：首先，它通过使用余弦相似度分析多语言词嵌入的一致性来评估语义相似性；其次，它通过命名实体识别和句子相似性任务来考察BLOOM-1.7B和Qwen2的语言结构理解能力；第三，它通过从高资源语言到低资源语言的情感分析和文本分类中的泛化性能评估跨语言知识迁移。该研究利用了语言探测、性能指标和可视化技术，从而为多语言自然语言处理（NLP）模型的优势和局限性提供了见解。这些发现旨在增强多语言NLP模型，确保对高资源和低资源语言都提供更好的支持，进而促进语言技术的包容性。
URL
https://arxiv.org/abs/2502.04269
PDF
-
Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis
2025-02-06 07:53:40Lin Yuan, Jun Xu, Honghao Gui, Mengshu Sun, Zhiqiang Zhang, Lei Liang, Jun ZhouAbstract
High-quality, large-scale instructions are crucial for aligning large language models (LLMs), however, there is a severe shortage of instruction in the field of natural language understanding (NLU). Previous works on constructing NLU instructions mainly focus on information extraction (IE), neglecting tasks such as machine reading comprehension, question answering, and text classification. Furthermore, the lack of diversity in the data has led to a decreased generalization ability of trained LLMs in other NLU tasks and a noticeable decline in the fundamental model's general capabilities. To address this issue, we propose Hum, a large-scale, high-quality synthetic instruction corpus for NLU tasks, designed to enhance the NLU capabilities of LLMs. Specifically, Hum includes IE (either close IE or open IE), machine reading comprehension, text classification, and instruction generalist tasks, thereby enriching task diversity. Additionally, we introduce a human-LLMs collaborative mechanism to synthesize instructions, which enriches instruction diversity by incorporating guidelines, preference rules, and format variants. We conduct extensive experiments on 5 NLU tasks and 28 general capability evaluation datasets for LLMs. Experimental results show that Hum enhances the NLU capabilities of six LLMs by an average of 3.1\%, with no significant decline observed in other general capabilities.
Abstract (translated)
高质量的大规模指令对于调整大规模语言模型（LLM）至关重要，然而，在自然语言理解（NLU）领域存在严重的指令短缺。以前构建NLU指令的工作主要集中在信息抽取（IE）上，忽略了诸如机器阅读理解、问答和文本分类等任务。此外，数据缺乏多样性导致经过训练的LLMs在其他NLU任务中的泛化能力下降，并且基础模型的基本能力也明显减弱。为了解决这个问题，我们提出了Hum，这是一个针对NLU任务的大规模高质量合成指令语料库，旨在增强LLM的NLU能力。具体来说，Hum包括信息抽取（无论是封闭式还是开放式）、机器阅读理解、文本分类和指令通才任务，从而丰富了任务多样性。此外，我们引入了一种人与LLMs协作机制来生成指令，通过融入指南、偏好规则和格式变体，丰富了指令的多样性。我们在5个NLU任务和28个针对LLM的一般能力评估数据集上进行了广泛的实验。实验结果显示，Hum使六种LLM的NLU能力平均提升了3.1%，并且在其他一般能力方面没有显著下降。
URL
https://arxiv.org/abs/2502.03843
PDF
-
Out-of-Distribution Detection using Synthetic Data Generation
2025-02-05 16:22:09Momin Abbas, Muneeza Azmat, Raya Horesh, Mikhail YurochkinAbstract
Distinguishing in- and out-of-distribution (OOD) inputs is crucial for reliable deployment of classification systems. However, OOD data is typically unavailable or difficult to collect, posing a significant challenge for accurate OOD detection. In this work, we present a method that harnesses the generative capabilities of Large Language Models (LLMs) to create high-quality synthetic OOD proxies, eliminating the dependency on any external OOD data source. We study the efficacy of our method on classical text classification tasks such as toxicity detection and sentiment classification as well as classification tasks arising in LLM development and deployment, such as training a reward model for RLHF and detecting misaligned generations. Extensive experiments on nine InD-OOD dataset pairs and various model sizes show that our approach dramatically lowers false positive rates (achieving a perfect zero in some cases) while maintaining high accuracy on in-distribution tasks, outperforming baseline methods by a significant margin.
Abstract (translated)
区分内分布（In-Distribution，ID）和外分布（Out-of-Distribution，OOD）输入对于分类系统的可靠部署至关重要。然而，OOD数据通常难以获取或收集，这给准确的OOD检测带来了重大挑战。在这项工作中，我们提出了一种方法，该方法利用大规模语言模型（LLMs）的生成能力来创建高质量的合成OOD代理，从而消除了对任何外部OOD数据源的依赖。我们在经典文本分类任务（如毒性检测和情感分类）以及在LLM开发和部署中出现的分类任务（例如为RLHF训练奖励模型和检测不一致生成）上研究了我们方法的有效性。在九组InD-OOD数据集对及各种规模模型上的大量实验表明，我们的方法显著降低了假阳性率（在某些情况下达到了完美的零），同时保持了高准确度的内分布任务表现，大幅超越了基线方法。
URL
https://arxiv.org/abs/2502.03323
PDF
-
Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attribution
2025-02-04 03:33:55Muhammad Umair Haider, Hammad Rizwan, Hassan Sajjad, Peizhong Ju, A. B. SiddiqueAbstract
Interpreting and controlling the internal mechanisms of large language models (LLMs) is crucial for improving their trustworthiness and utility. Recent efforts have primarily focused on identifying and manipulating neurons by establishing discrete mappings between neurons and semantic concepts. However, such mappings struggle to handle the inherent polysemanticity in LLMs, where individual neurons encode multiple, distinct concepts. This makes precise control challenging and complicates downstream interventions. Through an in-depth analysis of both encoder and decoder-based LLMs across multiple text classification datasets, we uncover that while individual neurons encode multiple concepts, their activation magnitudes vary across concepts in distinct, Gaussian-like patterns. Building on this insight, we introduce NeuronLens, a novel range-based interpretation and manipulation framework that provides a finer view of neuron activation distributions to localize concept attribution within a neuron. Extensive empirical evaluations demonstrate that NeuronLens significantly reduces unintended interference, while maintaining precise control for manipulation of targeted concepts, outperforming existing methods.
Abstract (translated)
解释和控制大型语言模型（LLMs）的内部机制对于提高其可靠性和实用性至关重要。近期的研究主要集中在通过建立神经元与语义概念之间的离散映射来识别和操纵神经元上。然而，这种映射方法难以处理大型语言模型中的固有多义性问题，即单个神经元编码多个不同的概念。这使得精确控制变得具有挑战性，并且增加了下游干预的复杂性。 通过对多种文本分类数据集上的基于编码器和解码器的LLMs进行深入分析，我们发现尽管单个神经元编码了多个概念，但它们在不同概念下的激活强度遵循着独特的、类似高斯分布的模式。基于这一洞察，我们引入了NeuronLens——一种新颖的基于范围的解释和操纵框架，该框架提供了对神经元激活分布更细致的视角，以便在单个神经元内定位具体的概念归属。 广泛的实证评估表明，与现有方法相比，NeuronLens显著减少了意外干扰，并保持了对目标概念精确控制的能力。
URL
https://arxiv.org/abs/2502.06809
PDF
-
DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning
2025-02-01 04:00:03Jiaxin Guo, C. L. Philip Chen, Shuzhen Li, Tong ZhangAbstract
Cold-start active learning (CSAL) selects valuable instances from an unlabeled dataset for manual annotation. It provides high-quality data at a low annotation cost for label-scarce text classification. However, existing CSAL methods overlook weak classes and hard representative examples, resulting in biased learning. To address these issues, this paper proposes a novel dual-diversity enhancing and uncertainty-aware (DEUCE) framework for CSAL. Specifically, DEUCE leverages a pretrained language model (PLM) to efficiently extract textual representations, class predictions, and predictive uncertainty. Then, it constructs a Dual-Neighbor Graph (DNG) to combine information on both textual diversity and class diversity, ensuring a balanced data distribution. It further propagates uncertainty information via density-based clustering to select hard representative instances. DEUCE performs well in selecting class-balanced and hard representative data by dual-diversity and informativeness. Experiments on six NLP datasets demonstrate the superiority and efficiency of DEUCE.
Abstract (translated)
冷启动主动学习（CSAL）从未标记的数据集中选择有价值的实例进行人工标注，为标签稀少的文本分类提供了高质量且成本低的数据。然而，现有的CSAL方法忽视了弱类别和难以代表的例子，导致了偏向性的学习问题。为了应对这些问题，本文提出了一种新颖的双多样性增强与不确定性感知（DEUCE）框架用于CSAL。具体来说，DEUCE利用预训练的语言模型（PLM）高效地提取文本表示、类别预测以及预测不确定性。接着，它构建了一个双邻域图（DNG），结合了文本多样性和类别多样性的信息，确保数据分布的均衡性。此外，通过基于密度的聚类传播不确定性信息，DEUCE选择出具有代表性的困难实例。 通过利用双重多样性与信息量，DEUCE在选择平衡且有代表性的样本方面表现出色。实验结果表明，在六个NLP数据集上，DEUCE展示了其优越性和效率。
URL
https://arxiv.org/abs/2502.00305
PDF
-
Memory-Efficient Fine-Tuning of Transformers via Token Selection
2025-01-31 00:43:50Antoine Simoulin, Namyong Park, Xiaoyi Liu, Grey YangAbstract
Fine-tuning provides an effective means to specialize pre-trained models for various downstream tasks. However, fine-tuning often incurs high memory overhead, especially for large transformer-based models, such as LLMs. While existing methods may reduce certain parts of the memory required for fine-tuning, they still require caching all intermediate activations computed in the forward pass to update weights during the backward pass. In this work, we develop TokenTune, a method to reduce memory usage, specifically the memory to store intermediate activations, in the fine-tuning of transformer-based models. During the backward pass, TokenTune approximates the gradient computation by backpropagating through just a subset of input tokens. Thus, with TokenTune, only a subset of intermediate activations are cached during the forward pass. Also, TokenTune can be easily combined with existing methods like LoRA, further reducing the memory cost. We evaluate our approach on pre-trained transformer models with up to billions of parameters, considering the performance on multiple downstream tasks such as text classification and question answering in a few-shot learning setup. Overall, TokenTune achieves performance on par with full fine-tuning or representative memory-efficient fine-tuning methods, while greatly reducing the memory footprint, especially when combined with other methods with complementary memory reduction mechanisms. We hope that our approach will facilitate the fine-tuning of large transformers, in specializing them for specific domains or co-training them with other neural components from a larger system. Our code is available at this https URL.
Abstract (translated)
微调提供了一种有效的方法，可以将预训练模型专门化为各种下游任务。然而，微调通常会产生较高的内存开销，尤其是对于基于转换器的大型语言模型（如LLM）。尽管现有方法可能减少了某些部分在微调过程中所需的内存，但它们仍然需要缓存前向传播计算的所有中间激活以更新反向传播过程中的权重。在此项工作中，我们开发了TokenTune方法，该方法旨在减少基于转换器模型的微调中用于存储中间激活的内存使用量。在反向传播期间，TokenTune通过仅对输入令牌的一个子集进行反向传播来近似梯度计算。因此，在前向传播过程中，只需要缓存一个部分的中间激活。此外，TokenTune可以轻松地与现有方法（如LoRA）结合使用，进一步减少内存成本。我们在具有数十亿参数的预训练转换器模型上评估了我们的方法，并考虑了在少量样本学习设置下的多个下游任务性能，例如文本分类和问答。总体而言，TokenTune实现了与完整微调或代表性内存高效微调方法相当的性能，同时大大减少了内存占用量，尤其是在与其他具有互补内存减少机制的方法结合使用时更是如此。我们希望我们的方法将有助于大型转换器模型的微调工作，以专门针对特定领域或在更大的系统中与其他神经组件进行联合训练。我们的代码可在以下链接获取：[此URL]。
URL
https://arxiv.org/abs/2501.18824
PDF
-
Semantic Consistency Regularization with Large Language Models for Semi-supervised Sentiment Analysis
2025-01-29 12:03:11Kunrong Li, Xinyu Liu, Zhen ChenAbstract
Accurate sentiment analysis of texts is crucial for a variety of applications, such as understanding customer feedback, monitoring market trends, and detecting public sentiment. However, manually annotating large sentiment corpora for supervised learning is labor-intensive and time-consuming. Therefore, it is essential and effective to develop a semi-supervised method for the sentiment analysis task. Although some methods have been proposed for semi-supervised text classification, they rely on the intrinsic information within the unlabeled data and the learning capability of the NLP model, which lack generalization ability to the sentiment analysis scenario and may prone to overfit. Inspired by the ability of pretrained Large Language Models (LLMs) in following instructions and generating coherent text, we propose a Semantic Consistency Regularization with Large Language Models (SCR) framework for semi-supervised sentiment analysis. We introduce two prompting strategies to semantically enhance unlabeled text using LLMs. The first is Entity-based Enhancement (SCR-EE), which involves extracting entities and numerical information, and querying the LLM to reconstruct the textual information. The second is Concept-based Enhancement (SCR-CE), which directly queries the LLM with the original sentence for semantic reconstruction. Subsequently, the LLM-augmented data is utilized for a consistency loss with confidence thresholding, which preserves high-quality agreement samples to provide additional supervision signals during training. Furthermore, to fully utilize the uncertain unlabeled data samples, we propose a class re-assembling strategy inspired by the class space shrinking theorem. Experiments show our method achieves remarkable performance over prior semi-supervised methods.
Abstract (translated)
准确的情感分析对于理解客户反馈、监控市场趋势和检测公众情感等众多应用至关重要。然而，为监督学习手动标注大量情感语料库既耗时又费力。因此，开发一种适用于情感分析任务的半监督方法是必要且有效的。虽然已经提出了一些用于半监督文本分类的方法，但这些方法依赖于未标记数据中的内在信息和NLP模型的学习能力，这在情感分析场景中缺乏泛化能力，并可能容易过拟合。受预训练大规模语言模型（LLM）遵循指令并生成连贯文本的能力的启发，我们提出了一种基于大规模语言模型的语义一致性正则化（SCR）框架用于半监督情感分析。我们介绍了两种提示策略，利用LLM来增强未标记文本的语义信息：一种是基于实体的增强（SCR-EE），涉及提取实体和数值信息，并查询LLM以重构文本信息；另一种是基于概念的增强（SCR-CE），直接使用原始句子查询LLM进行语义重构。随后，利用经过LLM扩充的数据应用一致性损失并采用置信度阈值筛选高质量的一致性样本，在训练过程中提供额外的监督信号。此外，为了充分利用不确定的未标记数据样本，我们提出了一种基于类别重组策略的方法，该方法受到类空间收缩定理的启发。实验结果表明，我们的方法在以前的半监督方法中取得了显著的效果。
URL
https://arxiv.org/abs/2501.17598
PDF
