Text_Classification
-
Guiding LLMs to Generate High-Fidelity and High-Quality Counterfactual Explanations for Text Classification
2025-03-06 14:15:07Van Bach Nguyen, Christin Seifert, J\"org Schl\"ottererAbstract
The need for interpretability in deep learning has driven interest in counterfactual explanations, which identify minimal changes to an instance that change a model's prediction. Current counterfactual (CF) generation methods require task-specific fine-tuning and produce low-quality text. Large Language Models (LLMs), though effective for high-quality text generation, struggle with label-flipping counterfactuals (i.e., counterfactuals that change the prediction) without fine-tuning. We introduce two simple classifier-guided approaches to support counterfactual generation by LLMs, eliminating the need for fine-tuning while preserving the strengths of LLMs. Despite their simplicity, our methods outperform state-of-the-art counterfactual generation methods and are effective across different LLMs, highlighting the benefits of guiding counterfactual generation by LLMs with classifier information. We further show that data augmentation by our generated CFs can improve a classifier's robustness. Our analysis reveals a critical issue in counterfactual generation by LLMs: LLMs rely on parametric knowledge rather than faithfully following the classifier.
Abstract (translated)
深度学习中对可解释性的需求推动了反事实解释的兴趣，这种解释能识别出使模型预测发生变化的实例最小修改。目前的反事实（CF）生成方法需要针对特定任务进行微调，并且会产生质量较低的文本。虽然大型语言模型（LLMs）在高质量文本生成方面非常有效，但它们在没有微调的情况下难以产生标签翻转型反事实（即改变预测结果的反事实）。我们介绍了两种简单的基于分类器引导的方法来支持由LLM生成反事实，这些方法消除了对微调的需求，同时保持了LLMs的优点。尽管这两种方法很简单，但它们超越了最先进的反事实生成方法，并且在不同的LLMs上都非常有效，这突显了通过分类器信息指导LLMs进行反事实生成的好处。我们进一步展示了由我们的生成CF进行的数据增强可以提高分类器的鲁棒性。我们的分析揭示了一个关键问题：LLMs依赖于参数知识而不是忠实遵循分类器。
URL
https://arxiv.org/abs/2503.04463
PDF
-
Large Language Models for Healthcare Text Classification: A Systematic Review
2025-03-03 04:16:13Hajar Sakai, Sarah S. LamAbstract
Large Language Models (LLMs) have fundamentally transformed approaches to Natural Language Processing (NLP) tasks across diverse domains. In healthcare, accurate and cost-efficient text classification is crucial, whether for clinical notes analysis, diagnosis coding, or any other task, and LLMs present promising potential. Text classification has always faced multiple challenges, including manual annotation for training, handling imbalanced data, and developing scalable approaches. With healthcare, additional challenges are added, particularly the critical need to preserve patients' data privacy and the complexity of the medical terminology. Numerous studies have been conducted to leverage LLMs for automated healthcare text classification and contrast the results with existing machine learning-based methods where embedding, annotation, and training are traditionally required. Existing systematic reviews about LLMs either do not specialize in text classification or do not focus on the healthcare domain. This research synthesizes and critically evaluates the current evidence found in the literature regarding the use of LLMs for text classification in a healthcare setting. Major databases (e.g., Google Scholar, Scopus, PubMed, Science Direct) and other resources were queried, which focused on the papers published between 2018 and 2024 within the framework of PRISMA guidelines, which resulted in 65 eligible research articles. These were categorized by text classification type (e.g., binary classification, multi-label classification), application (e.g., clinical decision support, public health and opinion analysis), methodology, type of healthcare text, and metrics used for evaluation and validation. This review reveals the existing gaps in the literature and suggests future research lines that can be investigated and explored.
Abstract (translated)
大型语言模型（LLMs）已经在不同领域的自然语言处理（NLP）任务中根本性地转变了方法。在医疗保健领域，无论是临床记录分析、诊断编码还是其他任何任务，准确且成本效益高的文本分类都至关重要，而LLMs展现出了巨大的潜力。然而，文本分类一直面临着诸如训练所需的手动注释、不平衡数据处理以及开发可扩展的方法等多重挑战。特别是在医疗保健中，还需要解决患者数据隐私保护这一关键问题及医学术语的复杂性带来的额外挑战。 许多研究致力于利用LLMs进行自动化的医疗文本分类，并将结果与传统基于机器学习方法（这些方法通常需要嵌入、注释和训练）相比较。现有的关于LLMs的系统综述要么不专门针对文本分类，要么不对准医疗保健领域。这项研究旨在综合并批判性地评估文献中有关在医疗环境下使用LLMs进行文本分类的现有证据。研究人员查询了主要数据库（例如Google Scholar、Scopus、PubMed、Science Direct）及其他资源，并按照PRISMA指南框架筛选出2018年至2024年间发表的相关论文，共找到65篇符合条件的研究文章。 这些研究被按文本分类类型（如二元分类或多标签分类）、应用领域（如临床决策支持、公共卫生和舆论分析）、方法论、医疗文本种类以及用于评估与验证的指标进行了分类。此综述揭示了文献中的现有空白，并提出了未来可以探索的研究方向。
URL
https://arxiv.org/abs/2503.01159
PDF
-
An evaluation of DeepSeek Models in Biomedical Natural Language Processing
2025-03-01 21:26:29Zaifu Zhan, Shuang Zhou, Huixue Zhou, Jiawen Deng, Yu Hou, Jeremy Yeung, Rui ZhangAbstract
The advancement of Large Language Models (LLMs) has significantly impacted biomedical Natural Language Processing (NLP), enhancing tasks such as named entity recognition, relation extraction, event extraction, and text classification. In this context, the DeepSeek series of models have shown promising potential in general NLP tasks, yet their capabilities in the biomedical domain remain underexplored. This study evaluates multiple DeepSeek models (Distilled-DeepSeek-R1 series and Deepseek-LLMs) across four key biomedical NLP tasks using 12 datasets, benchmarking them against state-of-the-art alternatives (Llama3-8B, Qwen2.5-7B, Mistral-7B, Phi-4-14B, Gemma-2-9B). Our results reveal that while DeepSeek models perform competitively in named entity recognition and text classification, challenges persist in event and relation extraction due to precision-recall trade-offs. We provide task-specific model recommendations and highlight future research directions. This evaluation underscores the strengths and limitations of DeepSeek models in biomedical NLP, guiding their future deployment and optimization.
Abstract (translated)
大型语言模型（LLMs）的发展在生物医学自然语言处理（NLP）领域产生了显著影响，提升了诸如命名实体识别、关系抽取、事件抽取和文本分类等任务的效果。在此背景下，DeepSeek系列模型在通用NLP任务中展示了巨大的潜力，但在生物医学领域的应用能力仍有待探索。本研究评估了多个DeepSeek模型（Distilled-DeepSeek-R1系列和Deepseek-LLMs）在四个关键的生物医学NLP任务中的表现，并使用12个数据集将它们与最先进的替代模型（如Llama3-8B、Qwen2.5-7B、Mistral-7B、Phi-4-14B、Gemma-2-9B）进行基准测试。我们的研究结果表明，尽管DeepSeek模型在命名实体识别和文本分类任务上表现出色，但在事件抽取和关系提取方面仍面临精度与召回率之间的权衡问题。我们提供了针对特定任务的模型推荐，并指出了未来的研究方向。这一评估强调了DeepSeek模型在生物医学NLP领域的优势和局限性，为它们未来的部署和优化提供指导。
URL
https://arxiv.org/abs/2503.00624
PDF
-
Active Few-Shot Learning for Text Classification
2025-02-26 03:30:13Saeed Ahmadnia, Arash Yousefi Jordehi, Mahsa Hosseini Khasheh Heyran, Seyed Abolghasem Mirroshandel, Owen Rambow, Cornelia CarageaAbstract
The rise of Large Language Models (LLMs) has boosted the use of Few-Shot Learning (FSL) methods in natural language processing, achieving acceptable performance even when working with limited training data. The goal of FSL is to effectively utilize a small number of annotated samples in the learning process. However, the performance of FSL suffers when unsuitable support samples are chosen. This problem arises due to the heavy reliance on a limited number of support samples, which hampers consistent performance improvement even when more support samples are added. To address this challenge, we propose an active learning-based instance selection mechanism that identifies effective support instances from the unlabeled pool and can work with different LLMs. Our experiments on five tasks show that our method frequently improves the performance of FSL. We make our implementation available on GitHub.
Abstract (translated)
大规模语言模型（LLMs）的兴起促进了在自然语言处理中使用少量样本学习（FSL）方法的应用，即使在数据量有限的情况下也能达到令人接受的效果。FSL的目标是有效利用少量标注样本进行学习过程。然而，当选择不合适的支持样本来训练时，FSL的表现会受到影响。这一问题源于对少量支持样本的过度依赖，这导致即便增加更多的支持样本也无法持续提升性能表现。为解决这一挑战，我们提出了一种基于主动学习的实例选择机制，该机制可以从未标记的数据池中识别出有效支持实例，并且可以与不同的LLMs协同工作。我们在五个任务上的实验表明，我们的方法能够显著提高FSL的表现。我们将代码开源在GitHub上。
URL
https://arxiv.org/abs/2502.18782
PDF
-
From Small to Large Language Models: Revisiting the Federalist Papers
2025-02-25 21:50:46So Won Jeong, Veronika RockovaAbstract
For a long time, the authorship of the Federalist Papers had been a subject of inquiry and debate, not only by linguists and historians but also by statisticians. In what was arguably the first Bayesian case study, Mosteller and Wallace (1963) provided the first statistical evidence for attributing all disputed papers to Madison. Our paper revisits this historical dataset but from a lens of modern language models, both small and large. We review some of the more popular Large Language Model (LLM) tools and examine them from a statistical point of view in the context of text classification. We investigate whether, without any attempt to fine-tune, the general embedding constructs can be useful for stylometry and attribution. We explain differences between various word/phrase embeddings and discuss how to aggregate them in a document. Contrary to our expectations, we exemplify that dimension expansion with word embeddings may not always be beneficial for attribution relative to dimension reduction with topic embeddings. Our experiments demonstrate that default LLM embeddings (even after manual fine-tuning) may not consistently improve authorship attribution accuracy. Instead, Bayesian analysis with topic embeddings trained on ``function words" yields superior out-of-sample classification performance. This suggests that traditional (small) statistical language models, with their interpretability and solid theoretical foundation, can offer significant advantages in authorship attribution tasks. The code used in this analysis is available at this http URL
Abstract (translated)
长期以来，《联邦党人论文》的作者身份一直是语言学家、历史学家乃至统计学家的研究和讨论对象。Mosteller 和 Wallace (1963) 的研究可以说是第一个贝叶斯案例研究，他们提供了将所有有争议的文章归于麦迪逊的第一份统计证据。本文重新审视了这个历史性数据集，但采用的是现代语言模型的视角，无论是小型还是大型模型。 我们回顾了一些较为流行的大型语言模型（LLM）工具，并从统计学的角度在文本分类的背景下对其进行研究。我们探讨了未经任何微调的情况下，通用嵌入结构是否可以用于风格计量和归因。我们将解释不同词/短语嵌入之间的差异，并讨论如何在文档中聚合它们。与我们的预期相反，我们举例说明了使用词嵌入进行维度扩展可能并不总是比使用主题嵌入进行维度缩减对作者归因更有益。 实验表明，默认的大型语言模型嵌入（即使经过手动微调）也不能一贯提高作者身份识别准确性。相比之下，基于“功能词”训练的主题嵌入与贝叶斯分析相结合，在外样本分类性能上表现出色。这表明传统的（小型）统计语言模型，凭借其可解释性和坚实的理论基础，可以在作者归因任务中提供显著优势。 本研究使用的代码可在以下链接获取：[此 HTTP URL]
URL
https://arxiv.org/abs/2503.01869
PDF
-
Enhancing Text Classification with a Novel Multi-Agent Collaboration Framework Leveraging BERT
2025-02-25 21:30:16Hediyeh Baban, Sai A Pidapar, Aashutosh Nema, Sichen LuAbstract
We introduce a novel multi-agent collaboration framework designed to enhance the accuracy and robustness of text classification models. Leveraging BERT as the primary classifier, our framework dynamically escalates low-confidence predictions to a specialized multi-agent system comprising Lexical, Contextual, Logic, Consensus, and Explainability agents. This collaborative approach allows for comprehensive analysis and consensus-driven decision-making, significantly improving classification performance across diverse text classification tasks. Empirical evaluations on benchmark datasets demonstrate that our framework achieves a 5.5% increase in accuracy compared to standard BERT-based classifiers, underscoring its effectiveness and academic novelty in advancing multi-agent systems within natural language processing.
Abstract (translated)
我们介绍了一种新颖的多代理协作框架，旨在提高文本分类模型的准确性和鲁棒性。该框架利用BERT作为主要分类器，并且能够动态地将低置信度预测提升到一个由词汇（Lexical）、上下文(Contextual)、逻辑(Logic)、共识(Consensus)和解释力(Explainability)代理组成的专门化多代理系统中进行处理。 这种协作方法允许进行全面分析和基于共识的决策制定，从而在各种文本分类任务上显著提高分类性能。我们在基准数据集上的实证评估表明，与标准BERT基线分类器相比，我们的框架实现了5.5%的准确率提升，这充分证明了其有效性和在自然语言处理领域推进多代理系统的学术新颖性。
URL
https://arxiv.org/abs/2502.18653
PDF
-
MAGE: Multi-Head Attention Guided Embeddings for Low Resource Sentiment Classification
2025-02-25 08:53:27Varun Vashisht, Samar Singh, Mihir Konduskar, Jaskaran Singh Walia, Vukosi MarivateAbstract
Due to the lack of quality data for low-resource Bantu languages, significant challenges are presented in text classification and other practical implementations. In this paper, we introduce an advanced model combining Language-Independent Data Augmentation (LiDA) with Multi-Head Attention based weighted embeddings to selectively enhance critical data points and improve text classification performance. This integration allows us to create robust data augmentation strategies that are effective across various linguistic contexts, ensuring that our model can handle the unique syntactic and semantic features of Bantu languages. This approach not only addresses the data scarcity issue but also sets a foundation for future research in low-resource language processing and classification tasks.
Abstract (translated)
由于缺乏高质量的数据，低资源班图语的语言文本分类和其他实际应用面临重大挑战。在本文中，我们引入了一种结合语言无关数据增强（LiDA）与基于多头注意力机制的加权嵌入的高级模型。这种组合允许我们选择性地强化关键数据点，并提高文本分类性能。该集成使得我们可以创建有效的数据增强策略，适用于各种语言环境，确保我们的模型能够处理班图语独特的语法和语义特征。这一方法不仅解决了数据稀缺的问题，还为低资源语言处理和分类任务的未来研究奠定了基础。
URL
https://arxiv.org/abs/2502.17987
PDF
-
Applying LLMs to Active Learning: Towards Cost-Efficient Cross-Task Text Classification without Manually Labeled Data
2025-02-24 06:43:19Yejian Zhang, Shingo TakadaAbstract
Machine learning-based classifiers have been used for text classification, such as sentiment analysis, news classification, and toxic comment classification. However, supervised machine learning models often require large amounts of labeled data for training, and manual annotation is both labor-intensive and requires domain-specific knowledge, leading to relatively high annotation costs. To address this issue, we propose an approach that integrates large language models (LLMs) into an active learning framework. Our approach combines the Robustly Optimized BERT Pretraining Approach (RoBERTa), Generative Pre-trained Transformer (GPT), and active learning, achieving high cross-task text classification performance without the need for any manually labeled data. Furthermore, compared to directly applying GPT for classification tasks, our approach retains over 93% of its classification performance while requiring only approximately 6% of the computational time and monetary cost, effectively balancing performance and resource efficiency. These findings provide new insights into the efficient utilization of LLMs and active learning algorithms in text classification tasks, paving the way for their broader application.
Abstract (translated)
基于机器学习的分类器已被用于文本分类，如情感分析、新闻分类和有毒评论分类。然而，监督式机器学习模型通常需要大量标注数据进行训练，并且手动标注既耗时又需要特定领域的知识，导致标注成本相对较高。为了解决这个问题，我们提出了一种将大型语言模型（LLM）集成到主动学习框架中的方法。我们的方法结合了稳健优化的BERT预训练方法（RoBERTa）、生成式预训练变换器（GPT）和主动学习，在无需任何人工标注数据的情况下实现了跨任务的高性能文本分类效果。此外，与直接应用GPT进行分类任务相比，我们的方法在保持超过93%分类性能的同时，仅需大约6%的计算时间和成本，有效平衡了性能和资源效率。这些发现为大型语言模型和主动学习算法在文本分类任务中的高效利用提供了新的见解，并为其更广泛的应用铺平道路。
URL
https://arxiv.org/abs/2502.16892
PDF
-
MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learning
2025-02-21 21:36:48Zaifu Zhan, Jun Wang, Shuang Zhou, Jiawen Deng, Rui ZhangAbstract
Objective: To optimize in-context learning in biomedical natural language processing by improving example selection. Methods: We introduce a novel multi-mode retrieval-augmented generation (MMRAG) framework, which integrates four retrieval strategies: (1) Random Mode, selecting examples arbitrarily; (2) Top Mode, retrieving the most relevant examples based on similarity; (3) Diversity Mode, ensuring variation in selected examples; and (4) Class Mode, selecting category-representative examples. This study evaluates MMRAG on three core biomedical NLP tasks: Named Entity Recognition (NER), Relation Extraction (RE), and Text Classification (TC). The datasets used include BC2GM for gene and protein mention recognition (NER), DDI for drug-drug interaction extraction (RE), GIT for general biomedical information extraction (RE), and HealthAdvice for health-related text classification (TC). The framework is tested with two large language models (Llama2-7B, Llama3-8B) and three retrievers (Contriever, MedCPT, BGE-Large) to assess performance across different retrieval strategies. Results: The results from the Random mode indicate that providing more examples in the prompt improves the model's generation performance. Meanwhile, Top mode and Diversity mode significantly outperform Random mode on the RE (DDI) task, achieving an F1 score of 0.9669, a 26.4% improvement. Among the three retrievers tested, Contriever outperformed the other two in a greater number of experiments. Additionally, Llama 2 and Llama 3 demonstrated varying capabilities across different tasks, with Llama 3 showing a clear advantage in handling NER tasks. Conclusion: MMRAG effectively enhances biomedical in-context learning by refining example selection, mitigating data scarcity issues, and demonstrating superior adaptability for NLP-driven healthcare applications.
Abstract (translated)
**目标:** 在生物医学自然语言处理中优化上下文学习，通过改进示例选择来提高性能。 **方法:** 我们引入了一种新颖的多模式检索增强生成（MMRAG）框架。该框架集成了四种检索策略：(1) 随机模式，随机选取例子；(2) 顶级模式，基于相似性提取最相关的例子；(3) 多样性模式，确保所选示例的变化性；以及 (4) 分类模式，选择具有代表性的类别实例。这项研究在三个核心生物医学NLP任务上评估了MMRAG：命名实体识别（NER）、关系抽取（RE）和文本分类（TC）。使用的数据集包括BC2GM用于基因和蛋白质提及识别（NER），DDI用于药物-药物相互作用提取（RE），GIT用于一般生物医学信息提取（RE），以及HealthAdvice用于与健康相关的文本分类（TC）。框架在两个大型语言模型（Llama2-7B，Llama3-8B）和三个检索器（Contriever，MedCPT，BGE-Large）上进行了测试，以评估不同检索策略下的性能。 **结果:** 随机模式的结果表明，在提示中提供更多的例子可以提高模型的生成表现。同时，顶级模式和多样性模式在关系抽取任务（DDI）中显著优于随机模式，取得了F1得分为0.9669的成绩，比随机模式提高了26.4%。在这三个检索器中，Contriever 在更多实验中表现更好于其他两个。此外，在不同的任务上，Llama 2和Llama 3表现出不同程度的能力，其中Llama 3在处理NER任务方面明显占优势。 **结论:** MMRAG通过优化示例选择来有效提高生物医学上下文学习的性能，减轻数据稀缺问题，并展示了对NLP驱动医疗应用的高度适应性。
URL
https://arxiv.org/abs/2502.15954
PDF
-
MHQA: A Diverse, Knowledge Intensive Mental Health Question Answering Challenge for Language Models
2025-02-21 12:37:58Suraj Racha, Prashant Joshi, Anshika Raman, Nikita Jangid, Mridul Sharma, Ganesh Ramakrishnan, Nirmal PunjabiAbstract
Mental health remains a challenging problem all over the world, with issues like depression, anxiety becoming increasingly common. Large Language Models (LLMs) have seen a vast application in healthcare, specifically in answering medical questions. However, there is a lack of standard benchmarking datasets for question answering (QA) in mental health. Our work presents a novel multiple choice dataset, MHQA (Mental Health Question Answering), for benchmarking Language models (LMs). Previous mental health datasets have focused primarily on text classification into specific labels or disorders. MHQA, on the other hand, presents question-answering for mental health focused on four key domains: anxiety, depression, trauma, and obsessive/compulsive issues, with diverse question types, namely, factoid, diagnostic, prognostic, and preventive. We use PubMed abstracts as the primary source for QA. We develop a rigorous pipeline for LLM-based identification of information from abstracts based on various selection criteria and converting it into QA pairs. Further, valid QA pairs are extracted based on post-hoc validation criteria. Overall, our MHQA dataset consists of 2,475 expert-verified gold standard instances called MHQA-gold and ~56.1k pairs pseudo labeled using external medical references. We report F1 scores on different LLMs along with few-shot and supervised fine-tuning experiments, further discussing the insights for the scores.
Abstract (translated)
心理健康仍然是全球面临的严峻问题，抑郁症和焦虑症等状况变得越来越普遍。大型语言模型（LLMs）在医疗领域得到了广泛的应用，尤其是在回答医学相关的问题方面。然而，在心理健康领域的问答基准数据集存在不足。我们的研究提出了一套新型的多选题数据集MHQA（心理健康问答），用于评估语言模型在这一特定领域的能力。 以往的心理健康数据集主要集中在文本分类到特定标签或疾病上，而MHQA则专注于四个关键领域的心理健康问题解答：焦虑、抑郁、创伤和强迫/侵扰障碍，并涵盖多种问题类型，包括事实性、诊断性、预后性和预防性。我们使用PubMed摘要作为问答的主要信息来源。 为了创建这一数据集，我们开发了一个严格的流程，利用大型语言模型从摘要中识别出符合各种选择标准的信息，并将其转化为问答对。此外，通过后期验证准则来提取有效的问题答案对。总的来说，我们的MHQA数据集包含2,475个经过专家验证的金标准实例（即MHQA-gold）和约56,100对使用外部医学参考进行伪标记的数据。 我们报告了不同语言模型在这一任务上的F1分数，并讨论了少量样本和监督微调实验中的得分见解。
URL
https://arxiv.org/abs/2502.15418
PDF
-
An Enhancement of Jiang, Z., et al.s Compression-Based Classification Algorithm Applied to News Article Categorization
2025-02-20 10:50:59Sean Lester C. Benavides, Cid Antonio F. Masapol, Jonathan C. Morano, Dan Michael A. CortezAbstract
This study enhances Jiang et al.'s compression-based classification algorithm by addressing its limitations in detecting semantic similarities between text documents. The proposed improvements focus on unigram extraction and optimized concatenation, eliminating reliance on entire document compression. By compressing extracted unigrams, the algorithm mitigates sliding window limitations inherent to gzip, improving compression efficiency and similarity detection. The optimized concatenation strategy replaces direct concatenation with the union of unigrams, reducing redundancy and enhancing the accuracy of Normalized Compression Distance (NCD) calculations. Experimental results across datasets of varying sizes and complexities demonstrate an average accuracy improvement of 5.73%, with gains of up to 11% on datasets containing longer documents. Notably, these improvements are more pronounced in datasets with high-label diversity and complex text structures. The methodology achieves these results while maintaining computational efficiency, making it suitable for resource-constrained environments. This study provides a robust, scalable solution for text classification, emphasizing lightweight preprocessing techniques to achieve efficient compression, which in turn enables more accurate classification.
Abstract (translated)
这项研究通过解决江等人基于压缩的分类算法在检测文本文档语义相似性方面的局限性，对其进行了改进。所提出的改进主要集中在单字符词（unigram）提取和优化拼接策略上，从而消除了对整个文档进行压缩的依赖。通过对提取出的单字符词进行压缩，该算法缓解了gzip滑动窗口固有的限制，提高了压缩效率并增强了相似性检测能力。优化后的拼接策略通过使用单字符词的并集取代直接拼接来减少冗余，并提升了标准化压缩距离（NCD）计算的准确性。 实验结果显示，在不同大小和复杂度的数据集中，平均准确率提高了5.73%，在包含较长文档的数据集中，这一改进幅度最高可达11%。值得注意的是，这种改进在标签多样性高且文本结构复杂的数据集上表现得更为显著。该方法在保持计算效率的同时实现了这些成果，使其适用于资源受限的环境。 这项研究提供了一种稳健、可扩展的解决方案来实现高效的文本分类，强调了通过轻量级预处理技术获得高效压缩的重要性，从而提高了分类准确性。
URL
https://arxiv.org/abs/2502.14444
PDF
-
Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models
2025-02-20 10:25:13Artem Vazhentsev, Lyudmila Rvanova, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem ShelmanovAbstract
Uncertainty quantification (UQ) is a prominent approach for eliciting truthful answers from large language models (LLMs). To date, information-based and consistency-based UQ have been the dominant UQ methods for text generation via LLMs. Density-based methods, despite being very effective for UQ in text classification with encoder-based models, have not been very successful with generative LLMs. In this work, we adapt Mahalanobis Distance (MD) - a well-established UQ technique in classification tasks - for text generation and introduce a new supervised UQ method. Our method extracts token embeddings from multiple layers of LLMs, computes MD scores for each token, and uses linear regression trained on these features to provide robust uncertainty scores. Through extensive experiments on eleven datasets, we demonstrate that our approach substantially improves over existing UQ methods, providing accurate and computationally efficient uncertainty scores for both sequence-level selective generation and claim-level fact-checking tasks. Our method also exhibits strong generalization to out-of-domain data, making it suitable for a wide range of LLM-based applications.
Abstract (translated)
不确定性量化（UQ）是一种从大型语言模型（LLM）中获取真实答案的重要方法。迄今为止，基于信息和一致性的方法是通过LLM进行文本生成的主要不确定性量化方法。尽管密度基方法在基于编码器的模型的文本分类任务中的不确定性量化方面非常有效，但它们在生成式LLM上并不成功。在这项工作中，我们改编了Mahalanobis距离（MD）——一种在分类任务中广泛应用的UQ技术，并引入了一种新的监督UQ方法。我们的方法从LLM的不同层提取token嵌入，为每个token计算MD分数，并使用基于这些特征训练的线性回归模型提供稳健的不确定性评分。通过在十一项数据集上的广泛实验，我们展示了该方法比现有的UQ方法有显著改进，能够为序列级的选择性生成和声明级别的事实核查任务提供准确且计算效率高的不确定性评分。此外，我们的方法还表现出强大的跨域泛化能力，适用于广泛的基于LLM的应用场景。
URL
https://arxiv.org/abs/2502.14427
PDF
-
QUAD-LLM-MLTC: Large Language Models Ensemble Learning for Healthcare Text Multi-Label Classification
2025-02-20 01:46:12Hajar Sakai, Sarah S. LamAbstract
The escalating volume of collected healthcare textual data presents a unique challenge for automated Multi-Label Text Classification (MLTC), which is primarily due to the scarcity of annotated texts for training and their nuanced nature. Traditional machine learning models often fail to fully capture the array of expressed topics. However, Large Language Models (LLMs) have demonstrated remarkable effectiveness across numerous Natural Language Processing (NLP) tasks in various domains, which show impressive computational efficiency and suitability for unsupervised learning through prompt engineering. Consequently, these LLMs promise an effective MLTC of medical narratives. However, when dealing with various labels, different prompts can be relevant depending on the topic. To address these challenges, the proposed approach, QUAD-LLM-MLTC, leverages the strengths of four LLMs: GPT-4o, BERT, PEGASUS, and BART. QUAD-LLM-MLTC operates in a sequential pipeline in which BERT extracts key tokens, PEGASUS augments textual data, GPT-4o classifies, and BART provides topics' assignment probabilities, which results in four classifications, all in a 0-shot setting. The outputs are then combined using ensemble learning and processed through a meta-classifier to produce the final MLTC result. The approach is evaluated using three samples of annotated texts, which contrast it with traditional and single-model methods. The results show significant improvements across the majority of the topics in the classification's F1 score and consistency (F1 and Micro-F1 scores of 78.17% and 80.16% with standard deviations of 0.025 and 0.011, respectively). This research advances MLTC using LLMs and provides an efficient and scalable solution to rapidly categorize healthcare-related text data without further training.
Abstract (translated)
收集到的医疗文本数据量的不断增加，为自动化多标签文本分类（MLTC）带来了独特的挑战。这一问题主要源于训练用注释文本的稀缺及其复杂性。传统的机器学习模型往往难以充分捕捉表达的主题范围。然而，在各种领域内，大型语言模型（LLMs）在众多自然语言处理（NLP）任务中展现了显著的效果，这些模型具有出色的计算效率，并且通过提示工程进行无监督学习表现出良好的适应性。因此，这些LLM为医疗叙事的MLTC提供了有效的解决方案。然而，在面对不同标签时，根据话题的不同，不同的提示可能是相关的。为了应对这些挑战，提出的QUAD-LLM-MLTC方法利用了四种大型语言模型（LLMs）的优势：GPT-4o、BERT、PEGASUS和BART。在该方法中，采用顺序流水线操作：首先由BERT提取关键标记，然后PEGASUS增强文本数据，接着GPT-4o进行分类，最后BART提供主题分配的概率，从而产生四类分类结果，并且所有这些都在零样本设置下完成（即无需进一步训练）。随后，通过集成学习将输出结合起来并通过元分类器处理以生成最终的MLTC结果。该方法利用三个注释文本样本进行了评估，将其与传统方法和单一模型方法进行对比。结果显示，在分类F1评分以及一致性方面，大多数话题都表现出显著改进（F1评分为78.17%，Micro-F1评分为80.16%，标准偏差分别为0.025和0.011）。这项研究推进了利用LLM进行MLTC的方法，并提供了一种有效且可扩展的解决方案，可以在无需进一步训练的情况下迅速分类医疗相关的文本数据。
URL
https://arxiv.org/abs/2502.14189
PDF
-
Self-Regularization with Latent Space Explanations for Controllable LLM-based Classification
2025-02-19 22:27:59Xuansheng Wu, Wenhao Yu, Xiaoming Zhai, Ninghao LiuAbstract
Modern text classification methods heavily rely on contextual embeddings from large language models (LLMs). Compared to human-engineered features, these embeddings provide automatic and effective representations for classification model training. However, they also introduce a challenge: we lose the ability to manually remove unintended features, such as sensitive or task-irrelevant features, to guarantee regulatory compliance or improve the generalizability of classification models. This limitation arises because LLM embeddings are opaque and difficult to interpret. In this paper, we propose a novel framework to identify and regularize unintended features in the LLM latent space. Specifically, we first pre-train a sparse autoencoder (SAE) to extract interpretable features from LLM latent spaces. To ensure the SAE can capture task-specific features, we further fine-tune it on task-specific datasets. In training the classification model, we propose a simple and effective regularizer, by minimizing the similarity between the classifier weights and the identified unintended feature, to remove the impacts of these unintended features toward classification. We evaluate the proposed framework on three real-world tasks, including toxic chat detection, reward modeling, and disease diagnosis. Results show that the proposed framework can significantly improve the classifier's generalizability by regularizing those features that are not semantically correlated to each task. This work pioneers controllable text classification on LLM latent spaces by leveraging interpreted features to address generalizability, fairness, and privacy challenges. We will release our code and data once accepted.
Abstract (translated)
现代文本分类方法高度依赖于大型语言模型（LLM）生成的上下文嵌入。与人工设计的特征相比，这些嵌入能自动且有效地为分类模型训练提供表示方式。然而，这也带来了一个挑战：我们失去了手动移除敏感或任务无关特征的能力，以确保符合监管要求或提升分类模型的一般适用性。这种限制出现的原因是LLM嵌入具有不透明性和难以解释的特点。在本文中，我们提出了一种新颖的框架，用于识别和规范化LLM潜在空间中的非意图特征。具体来说，我们首先预训练了一个稀疏自动编码器（SAE），以从LLM潜在空间中提取可解释的特征。为了确保SAE能够捕捉到特定任务的相关特征，我们在特定任务的数据集上进一步微调了它。在训练分类模型时，我们提出了一种简单且有效的正则化方法：通过最小化分类器权重与识别出的非意图特征之间的相似性来移除这些非意图特征对分类的影响。我们在三个现实世界任务中评估了所提出的框架，包括有毒聊天检测、奖励建模和疾病诊断。结果表明，该框架能够显著提高分类器的一般适用性，通过规范化那些在语义上与每个特定任务无关的特征。这项工作开创了基于解释特征解决大型语言模型潜在空间中的可控制文本分类问题的方法，以应对一般适用性、公平性和隐私方面的挑战。一旦被接受，我们将发布我们的代码和数据集。
URL
https://arxiv.org/abs/2502.14133
PDF
-
Language Models Can Predict Their Own Behavior
2025-02-18 23:13:16Dhananjay Ashok, Jonathan MayAbstract
Autoregressive Language Models output text by sequentially predicting the next token to generate, with modern methods like Chain-of-Thought (CoT) prompting achieving state-of-the-art reasoning capabilities by scaling the number of generated tokens. However, are there times when we can infer how the model will behave (e.g. abstain from answering a question) early in the computation, making generation unnecessary? We show that internal representation of input tokens alone can often precisely predict, not just the next token, but eventual behavior over the entire output sequence. We leverage this capacity and learn probes on internal states to create early warning (and exit) systems. Specifically, if the probes can confidently estimate the way the LM is going to behave, then the system will avoid generating tokens altogether and return the estimated behavior instead. On 27 text classification datasets spanning five different tasks, we apply this method to estimate the eventual answer of an LM under CoT prompting, reducing inference costs by 65% (average) while suffering an accuracy loss of no more than 1.4% (worst case). We demonstrate the potential of this method to pre-emptively identify when a model will abstain from answering a question, fail to follow output format specifications, or give a low-confidence response. We explore the limits of this capability, showing that probes generalize to unseen datasets, but perform worse when LM outputs are longer and struggle to predict properties that require access to knowledge that the models themselves lack. Encouragingly, performance scales with model size, suggesting applicability to the largest of models
Abstract (translated)
这项研究探讨了自动回归语言模型的行为预测能力。这类模型通过顺序地预测下一个标记来生成文本，现代方法如链式思维（CoT）提示法通过增加生成的标记数量来实现最先进的推理能力。然而，有时候我们可以在计算早期就推断出模型如何行为（例如，放弃回答一个问题），从而使得生成过程变得不必要。 研究人员发现，仅凭输入标记的内部表示就可以准确地预测下一个标记以及整个输出序列的行为。他们利用这一点开发了在内部状态上的探测器来创建预警系统，并且如果这些探测器能够自信地估计出语言模型（LM）将如何行为，则该系统可以完全避免生成标记并直接返回预计的结果。 这项研究在一个包含五个不同任务的27个文本分类数据集上进行了应用，通过这种方法，在CoT提示法下预测LM最终答案时，推理成本减少了65%（平均），而准确率损失不超过1.4%（最坏情况）。研究表明，该方法能够预先识别模型将何时放弃回答问题、无法遵循输出格式规范或给出低置信度响应。 研究人员还探讨了这种方法的能力极限。探测器可以泛化到未见过的数据集上，但在LM输出较长时性能下降，并且难以预测需要访问模型自身缺乏的知识的属性。令人鼓舞的是，该方法的表现随着模型规模的增长而提高，表明它适用于最大的模型。 总的来说，这项研究提供了一种新的途径来优化大型语言模型的计算效率和资源利用，同时保持高准确率。这种方法不仅可以降低推理成本，还可以改进模型在特定任务上的表现，并且预示着未来可能有更多关于如何更有效地使用大规模语言模型的研究方向。
URL
https://arxiv.org/abs/2502.13329
PDF
-
Performance Evaluation of Sentiment Analysis on Text and Emoji Data Using End-to-End, Transfer Learning, Distributed and Explainable AI Models
2025-02-18 20:58:37Sirisha Velampalli, Chandrashekar Muniyappa, Ashutosh SaxenaAbstract
Emojis are being frequently used in todays digital world to express from simple to complex thoughts more than ever before. Hence, they are also being used in sentiment analysis and targeted marketing campaigns. In this work, we performed sentiment analysis of Tweets as well as on emoji dataset from the Kaggle. Since tweets are sentences we have used Universal Sentence Encoder (USE) and Sentence Bidirectional Encoder Representations from Transformers (SBERT) end-to-end sentence embedding models to generate the embeddings which are used to train the Standard fully connected Neural Networks (NN), and LSTM NN models. We observe the text classification accuracy was almost the same for both the models around 98 percent. On the contrary, when the validation set was built using emojis that were not present in the training set then the accuracy of both the models reduced drastically to 70 percent. In addition, the models were also trained using the distributed training approach instead of a traditional singlethreaded model for better scalability. Using the distributed training approach, we were able to reduce the run-time by roughly 15% without compromising on accuracy. Finally, as part of explainable AI the Shap algorithm was used to explain the model behaviour and check for model biases for the given feature set.
Abstract (translated)
在当今的数字世界中，表情符号（Emojis）被频繁地用于表达从简单到复杂的思想，比以往任何时候都更多。因此，它们也被应用于情感分析和定向市场营销活动中。在这项工作中，我们对来自Kaggle的推特数据集以及Emoji数据集进行了情感分析。由于推文是句子形式，我们使用了通用句子编码器（Universal Sentence Encoder, USE）及句向量双向变换模型（Sentence Bidirectional Encoder Representations from Transformers, SBERT）生成嵌入，并用于训练标准全连接神经网络（NN）和长短期记忆神经网络（LSTM NN）模型。我们观察到，对于这两种模型而言，文本分类的准确率几乎相同，约为98%。 然而，当使用不在训练集中的表情符号构建验证集时，两个模型的准确性都大幅下降至70%左右。此外，为了提高可扩展性，我们将分布式培训方法应用于这些模型，而不是传统的单线程模型进行训练。采用这种分布式训练方式后，我们在不牺牲准确性的前提下将运行时间减少了大约15%。 最后，在解释人工智能（Explainable AI）的框架内，我们使用Shap算法来解释模型行为，并检查在给定特征集下的模型偏见问题。
URL
https://arxiv.org/abs/2502.13278
PDF
-
Private Text Generation by Seeding Large Language Model Prompts
2025-02-18 16:50:38Supriya Nagesh, Justin Y. Chen, Nina Mishra, Tal WagnerAbstract
We explore how private synthetic text can be generated by suitably prompting a large language model (LLM). This addresses a challenge for organizations like hospitals, which hold sensitive text data like patient medical records, and wish to share it in order to train machine learning models for medical tasks, while preserving patient privacy. Methods that rely on training or finetuning a model may be out of reach, either due to API limits of third-party LLMs, or due to ethical and legal prohibitions on sharing the private data with the LLM itself. We propose Differentially Private Keyphrase Prompt Seeding (DP-KPS), a method that generates a private synthetic text corpus from a sensitive input corpus, by accessing an LLM only through privatized prompts. It is based on seeding the prompts with private samples from a distribution over phrase embeddings, thus capturing the input corpus while achieving requisite output diversity and maintaining differential privacy. We evaluate DP-KPS on downstream ML text classification tasks, and show that the corpora it generates preserve much of the predictive power of the original ones. Our findings offer hope that institutions can reap ML insights by privately sharing data with simple prompts and little compute.
Abstract (translated)
我们研究了通过适当提示大型语言模型（LLM）生成私人合成文本的方法。这解决了像医院这样的组织面临的挑战，这些组织持有敏感的文本数据，如患者医疗记录，并希望在保护患者隐私的同时分享这些数据以训练用于医学任务的机器学习模型。依赖于对模型进行训练或微调的方法可能难以实现，要么是因为第三方LLM的API限制，要么是由于将私有数据与LLM本身共享时存在的伦理和法律禁令。我们提出了一种名为“差异隐私关键词提示播种”（DP-KPS）的方法，该方法通过仅使用私有化提示访问LLM来从敏感输入语料库生成私人合成文本语料库。它基于使用来自短语嵌入分布的私有样本对提示进行播种，从而捕获输入语料库的同时实现所需的输出多样性并保持差分隐私。我们在下游ML文本分类任务上评估了DP-KPS，并展示了其所生成的语料库保留了原始数据大部分的预测能力。我们的发现表明，机构可以通过简单的提示和少量计算私下分享数据来获取机器学习见解，这为未来的应用提供了希望。
URL
https://arxiv.org/abs/2502.13193
PDF
-
A Survey of Text Classification Under Class Distribution Shift
2025-02-18 15:46:54Adriana Valentina Costache, Silviu Florin Gheorghe, Eduard Gabriel Poesina, Paul Irofti, Radu Tudor IonescuAbstract
The basic underlying assumption of machine learning (ML) models is that the training and test data are sampled from the same distribution. However, in daily practice, this assumption is often broken, i.e.~the distribution of the test data changes over time, which hinders the application of conventional ML models. One domain where the distribution shift naturally occurs is text classification, since people always find new topics to discuss. To this end, we survey research articles studying open-set text classification and related tasks. We divide the methods in this area based on the constraints that define the kind of distribution shift and the corresponding problem formulation, i.e.~learning with the Universum, zero-shot learning, and open-set learning. We next discuss the predominant mitigation approaches for each problem setup. Finally, we identify several future work directions, aiming to push the boundaries beyond the state of the art. Interestingly, we find that continual learning can solve many of the issues caused by the shifting class distribution. We maintain a list of relevant papers at this https URL.
Abstract (translated)
机器学习（ML）模型的基本假设是训练数据和测试数据来自相同的分布。然而，在实际应用中，这一假设经常被打破，即测试数据的分布会随时间发生变化，这限制了传统机器学习模型的应用范围。在文本分类领域，这种分布变化自然发生，因为人们总是会讨论新的话题。为此，我们总结了研究开放集文本分类及相关任务的研究文章。我们将该领域的方法根据定义分布变化类型的约束以及相应的问题表述进行分类，即基于全集的学习、零样本学习和开放集学习。接下来，我们将讨论每种问题设置下的主要缓解措施。最后，我们确定了一些未来的工作方向，旨在推动超越当前技术水平的边界。有趣的是，我们发现连续学习可以解决由类别分布变化引起许多问题。我们在[这里](https://this.url/)维护了一个相关论文列表。
URL
https://arxiv.org/abs/2502.12965
PDF
-
Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text
2025-02-18 15:36:16Andrei Jarca, Florinel Alin Croitoru, Radu Tudor IonescuAbstract
Masked language modeling has become a widely adopted unsupervised technique to pre-train language models. However, the process of selecting tokens for masking is random, and the percentage of masked tokens is typically fixed for the entire training process. In this paper, we propose to adjust the masking ratio and to decide which tokens to mask based on a novel task-informed anti-curriculum learning scheme. First, we harness task-specific knowledge about useful and harmful tokens in order to determine which tokens to mask. Second, we propose a cyclic decaying masking ratio, which corresponds to an anti-curriculum schedule (from hard to easy). We exemplify our novel task-informed anti-curriculum by masking (TIACBM) approach across three diverse downstream tasks: sentiment analysis, text classification by topic, and authorship attribution. Our findings suggest that TIACBM enhances the ability of the model to focus on key task-relevant features, contributing to statistically significant performance gains across tasks. We release our code at this https URL.
Abstract (translated)
遮罩语言模型已成为预训练语言模型中广泛采用的无监督技术。然而，选择进行遮蔽的标记的过程是随机的，并且在整个训练过程中被遮蔽标记的比例通常是固定的。在本文中，我们提出调整遮蔽比例并根据一种新的任务相关反课程学习方案来决定要遮蔽哪些标记。首先，我们利用特定于任务的知识来确定哪些标记是有用的或有害的，从而决定要遮蔽哪些标记。其次，我们提出了一个周期性衰减的遮蔽比率，这对应于从难到易的反课程安排。 我们通过三种不同的下游任务展示了我们的新型任务相关反课程方法：情感分析、按主题分类和作者归属。实验结果表明，TIACBM（Task-Informed Anti-Curriculum Based Masking）能够增强模型聚焦于关键的任务相关信息的能力，在各个任务上带来了统计显著的性能提升。我们在提供的链接中发布了我们的代码。 原文链接： [此处应插入实际的URL](this https URL)
URL
https://arxiv.org/abs/2502.12953
PDF
-
AdaSplash: Adaptive Sparse Flash Attention
2025-02-17 17:56:23Nuno Gon\c{c}alves, Marcos Treviso, Andr\'e F. T. MartinsAbstract
The computational cost of softmax-based attention in transformers limits their applicability to long-context tasks. Adaptive sparsity, of which $\alpha$-entmax attention is an example, offers a flexible data-dependent alternative, but existing implementations are inefficient and do not leverage the sparsity to obtain runtime and memory gains. In this work, we propose AdaSplash, which combines the efficiency of GPU-optimized algorithms with the sparsity benefits of $\alpha$-entmax. We first introduce a hybrid Halley-bisection algorithm, resulting in a 7-fold reduction in the number of iterations needed to compute the $\alpha$-entmax transformation. Then, we implement custom Triton kernels to efficiently handle adaptive sparsity. Experiments with RoBERTa and ModernBERT for text classification and single-vector retrieval, along with GPT-2 for language modeling, show that our method achieves substantial improvements in runtime and memory efficiency compared to existing $\alpha$-entmax implementations. It approaches -- and in some cases surpasses -- the efficiency of highly optimized softmax implementations like FlashAttention-2, enabling long-context training while maintaining strong task performance.
Abstract (translated)
基于softmax的注意力机制在变压器中的计算成本限制了其在长上下文任务中的适用性。自适应稀疏性，例如$\alpha$-entmax注意就是此类方法的一个例子，提供了一种灵活的数据依赖性替代方案，但现有的实现效率低下，并没有利用稀疏性来获得运行时间和内存的节省。在这项工作中，我们提出了AdaSplash，它结合了GPU优化算法的效率与$\alpha$-entmax的稀疏性优势。首先，我们引入了一个混合Halley-二分法算法，使得计算$\alpha$-entmax变换所需的迭代次数减少了7倍。然后，我们通过自定义Triton内核有效地处理适应性稀疏性问题。使用RoBERTa和ModernBERT进行文本分类和单向量检索的实验，以及GPT-2用于语言建模，证明了我们的方法在运行时间和内存效率方面相比现有的$\alpha$-entmax实现取得了显著改进。我们的方法接近于——并在某些情况下超过了——高度优化的softmax实现（如FlashAttention-2）的效率，这使得能够进行长上下文训练的同时保持强大的任务性能。
URL
https://arxiv.org/abs/2502.12082
PDF
