Text_Classification
-
mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks
2025-06-10 03:15:08Luel Hagos Beyene, Vivek Verma, Min Ma, Jesujoba O. Alabi, Fabian David Schmidt, Joyce Nakatumba-Nabende, David Ifeoluwa AdelaniAbstract
Large Language models (LLMs) have demonstrated impressive performance on a wide range of tasks, including in multimodal settings such as speech. However, their evaluation is often limited to English and a few high-resource languages. For low-resource languages, there is no standardized evaluation benchmark. In this paper, we address this gap by introducing mSTEB, a new benchmark to evaluate the performance of LLMs on a wide range of tasks covering language identification, text classification, question answering, and translation tasks on both speech and text modalities. We evaluated the performance of leading LLMs such as Gemini 2.0 Flash and GPT-4o (Audio) and state-of-the-art open models such as Qwen 2 Audio and Gemma 3 27B. Our evaluation shows a wide gap in performance between high-resource and low-resource languages, especially for languages spoken in Africa and Americas/Oceania. Our findings show that more investment is needed to address their under-representation in LLMs coverage.
Abstract (translated)
大型语言模型（LLMs）在包括语音在内的多种模态任务中展现出了卓越的性能。然而，这些模型的评估通常仅限于英语及其他几种资源丰富的语言。对于资源匮乏的语言，缺乏标准化的评估基准。为解决这一问题，本文引入了mSTEB这一新的评估基准，用以衡量LLMs在语言识别、文本分类、问答和翻译任务上的表现，覆盖语音和文本模态的广泛任务。我们对领先的大规模模型（如Gemini 2.0 Flash 和 GPT-4o (Audio)）以及最先进的开源模型（如Qwen 2 Audio 和 Gemma 3 27B）进行了评估。我们的研究结果表明，在资源丰富与匮乏的语言之间，尤其是在非洲和美洲/大洋洲使用的语言上，LLMs的表现存在显著差距。这一发现表明，需要更多的投资来解决这些语言在LLMs覆盖中的不足问题。
URL
https://arxiv.org/abs/2506.08400
PDF
-
MultiMatch: Multihead Consistency Regularization Matching for Semi-Supervised Text Classification
2025-06-09 14:27:47Iustin Sirbu (National University of Science and Technology POLITEHNICA Bucharest), Robert-Adrian Popovici (National University of Science and Technology POLITEHNICA Bucharest), Cornelia Caragea (University of Illinois Chicago), Stefan Trausan-Matu (National University of Science and Technology POLITEHNICA Bucharest), Traian Rebedea (National University of Science and Technology POLITEHNICA Bucharest, NVIDIA)Abstract
We introduce MultiMatch, a novel semi-supervised learning (SSL) algorithm combining the paradigms of co-training and consistency regularization with pseudo-labeling. At its core, MultiMatch features a three-fold pseudo-label weighting module designed for three key purposes: selecting and filtering pseudo-labels based on head agreement and model confidence, and weighting them according to the perceived classification difficulty. This novel module enhances and unifies three existing techniques -- heads agreement from Multihead Co-training, self-adaptive thresholds from FreeMatch, and Average Pseudo-Margins from MarginMatch -- resulting in a holistic approach that improves robustness and performance in SSL settings. Experimental results on benchmark datasets highlight the superior performance of MultiMatch, achieving state-of-the-art results on 9 out of 10 setups from 5 natural language processing datasets and ranking first according to the Friedman test among 19 methods. Furthermore, MultiMatch demonstrates exceptional robustness in highly imbalanced settings, outperforming the second-best approach by 3.26% -- and data imbalance is a key factor for many text classification tasks.
Abstract (translated)
我们介绍了MultiMatch，这是一种新颖的半监督学习（SSL）算法，结合了联合训练和一致性正则化与伪标签生成的范式。在核心部分，MultiMatch包含一个三重伪标签加权模块，旨在实现三个关键目标：根据头部一致性和模型置信度选择并过滤伪标签，并根据感知分类难度对它们进行加权。这一创新模块增强了并统一了三种现有技术——来自Multihead Co-training的头部一致性、FreeMatch中的自适应阈值以及MarginMatch中的平均伪边界，从而形成了一种全面提升半监督学习场景中鲁棒性和性能的整体方法。在基准数据集上的实验结果突显了MultiMatch的优越表现，在五个自然语言处理数据集中十个设置中有九个达到了最先进的水平，并且根据Friedman检验在19种方法中排名第一。此外，MultiMatch展示了其在高度不平衡环境中的卓越稳健性，超越了第二最佳的方法3.26%——而数据不平衡是许多文本分类任务的关键因素。
URL
https://arxiv.org/abs/2506.07801
PDF
-
CTDGSI: A comprehensive exploitation of instance selection methods for automatic text classification. VII Concurso de Teses, Dissertac{c}~oes e Trabalhos de Graduac{c}~ao em SI -- XXI Simp'osio Brasileiro de Sistemas de Informac{c}~ao
2025-06-08 14:34:57Washington Cunha, Leonardo Rocha, Marcos Andr\'e Gon\c{c}alvesAbstract
Progress in Natural Language Processing (NLP) has been dictated by the rule of more: more data, more computing power and more complexity, best exemplified by the Large Language Models. However, training (or fine-tuning) large dense models for specific applications usually requires significant amounts of computing resources. This \textbf{Ph.D. dissertation} focuses on an under-investi\-gated NLP data engineering technique, whose potential is enormous in the current scenario known as Instance Selection (IS). The IS goal is to reduce the training set size by removing noisy or redundant instances while maintaining the effectiveness of the trained models and reducing the training process cost. We provide a comprehensive and scientifically sound comparison of IS methods applied to an essential NLP task -- Automatic Text Classification (ATC), considering several classification solutions and many datasets. Our findings reveal a significant untapped potential for IS solutions. We also propose two novel IS solutions that are noise-oriented and redundancy-aware, specifically designed for large datasets and transformer architectures. Our final solution achieved an average reduction of 41\% in training sets, while maintaining the same levels of effectiveness in all datasets. Importantly, our solutions demonstrated speedup improvements of 1.67x (up to 2.46x), making them scalable for datasets with hundreds of thousands of documents.
Abstract (translated)
自然语言处理（NLP）的进步一直遵循“越多越好”的原则：更多的数据、更强的计算能力和更复杂的模型，这一点在大型语言模型中得到了最佳体现。然而，为特定应用训练（或微调）大而密集的模型通常需要大量的计算资源。本**博士论文**重点研究了一种未被充分探索的NLP数据工程技术——实例选择（IS），其在当前背景下具有巨大的潜力。IS的目标是通过移除噪声或冗余样本减少训练集大小，同时保持训练出的模型的有效性并降低训练成本。 我们在几种分类解决方案和许多数据集上对IS方法进行了全面而科学的比较，特别是针对NLP中的关键任务——自动文本分类（ATC）。我们的研究发现揭示了IS解决方案的巨大未开发潜力。此外，我们还提出了两种新的噪声导向和冗余感知的IS方案，专门设计用于大规模数据集和Transformer架构。 最终，我们的解决方案在所有数据集中平均减少了41%的训练集大小，同时保持相同的有效性水平。重要的是，我们的方法展示了高达1.67倍（最高达2.46倍）的速度提升，使其能够扩展到包含数十万文档的数据集上。
URL
https://arxiv.org/abs/2506.07169
PDF
-
Label-semantics Aware Generative Approach for Domain-Agnostic Multilabel Classification
2025-06-07 14:07:07Subhendu Khatuya, Shashwat Naidu, Saptarshi Ghosh, Pawan Goyal, Niloy GangulyAbstract
The explosion of textual data has made manual document classification increasingly challenging. To address this, we introduce a robust, efficient domain-agnostic generative model framework for multi-label text classification. Instead of treating labels as mere atomic symbols, our approach utilizes predefined label descriptions and is trained to generate these descriptions based on the input text. During inference, the generated descriptions are matched to the pre-defined labels using a finetuned sentence transformer. We integrate this with a dual-objective loss function, combining cross-entropy loss and cosine similarity of the generated sentences with the predefined target descriptions, ensuring both semantic alignment and accuracy. Our proposed model LAGAMC stands out for its parameter efficiency and versatility across diverse datasets, making it well-suited for practical applications. We demonstrate the effectiveness of our proposed model by achieving new state-of-the-art performances across all evaluated datasets, surpassing several strong baselines. We achieve improvements of 13.94% in Micro-F1 and 24.85% in Macro-F1 compared to the closest baseline across all datasets.
Abstract (translated)
文本数据的爆炸式增长使得手动文档分类变得越来越具有挑战性。为了解决这一问题，我们引入了一个健壮、高效且领域无关的生成模型框架，用于多标签文本分类。与将标签视为单纯原子符号的做法不同，我们的方法利用预定义的标签描述，并训练模型根据输入文本生成这些描述。在推理阶段，通过使用经过微调的句子变换器将生成的描述匹配到预先定义的标签上。我们集成了一个双重目标损失函数，结合了交叉熵损失和生成句与预定义的目标描述之间的余弦相似度，以确保语义的一致性和准确性。 我们的模型LAGAMC（Label Description Guided Attention Model for Multi-label Classification）以其参数效率和跨多种数据集的通用性而脱颖而出，非常适合实际应用。我们通过在所有评估的数据集中实现新的最先进性能，并超越了多个强大的基线，证明了所提模型的有效性。与最近的最佳方法相比，我们在所有数据集上分别实现了微平均F1分数提高了13.94%，宏平均F1分数提高了24.85%的成绩。
URL
https://arxiv.org/abs/2506.06806
PDF
-
LTG at SemEval-2025 Task 10: Optimizing Context for Classification of Narrative Roles
2025-06-06 10:58:45Egil R{\o}nningstad, Gaurav NegiAbstract
Our contribution to the SemEval 2025 shared task 10, subtask 1 on entity framing, tackles the challenge of providing the necessary segments from longer documents as context for classification with a masked language model. We show that a simple entity-oriented heuristics for context selection can enable text classification using models with limited context window. Our context selection approach and the XLM-RoBERTa language model is on par with, or outperforms, Supervised Fine-Tuning with larger generative language models.
Abstract (translated)
我们在 SemEval 2025 共享任务 10 的子任务 1（实体框架）中的贡献，旨在解决从较长文档中提供必要段落作为上下文以供带有掩码语言模型进行分类的挑战。我们证明了一个简单的基于实体的上下文选择启发式方法可以利用具有有限上下文窗口的模型来进行文本分类。我们的上下文选择方法和 XLM-RoBERTa 语言模型在性能上可与使用更大生成型语言模型的监督微调方法相媲美，甚至超越后者。
URL
https://arxiv.org/abs/2506.05976
PDF
-
Hierarchical Text Classification Using Contrastive Learning Informed Path Guided Hierarchy
2025-06-04 18:51:34Neeraj Agrawal, Saurabh Kumar, Priyanka Bhatt, Tanishka AgarwalAbstract
Hierarchical Text Classification (HTC) has recently gained traction given the ability to handle complex label hierarchy. This has found applications in domains like E- commerce, customer care and medicine industry among other real-world applications. Existing HTC models either encode label hierarchy separately and mix it with text encoding or guide the label hierarchy structure in the text encoder. Both approaches capture different characteristics of label hierarchy and are complementary to each other. In this paper, we propose a Hierarchical Text Classification using Contrastive Learning Informed Path guided hierarchy (HTC-CLIP), which learns hierarchy-aware text representation and text informed path guided hierarchy representation using contrastive learning. During the training of HTC-CLIP, we learn two different sets of class probabilities distributions and during inference, we use the pooled output of both probabilities for each class to get the best of both representations. Our results show that the two previous approaches can be effectively combined into one architecture to achieve improved performance. Tests on two public benchmark datasets showed an improvement of 0.99 - 2.37% in Macro F1 score using HTC-CLIP over the existing state-of-the-art models.
Abstract (translated)
层次文本分类（HTC）最近因其处理复杂标签层级的能力而受到关注，这在电子商务、客户服务和医药行业等众多实际应用中得到了广泛应用。现有的HTC模型要么独立编码标签层级并将其与文本编码混合，要么指导文本编码器中的标签层级结构。这两种方法分别捕捉了标签层级的不同特征，并且互为补充。 本文提出了一种使用对比学习引导路径的层次文本分类（HTC-CLIP）方法，该方法利用对比学习来学习层次感知的文本表示和基于文本的路径引导层次表示。在HTC-CLIP的训练过程中，我们学习两组不同的类别概率分布，并在推理阶段结合这两种概率分布的结果以获得最佳表现。 实验结果表明，将先前提出的两种方法有效整合到一个架构中可以实现性能提升。在两个公开基准数据集上的测试显示，与现有最先进的模型相比，HTC-CLIP的Macro F1得分提高了0.99%至2.37%。
URL
https://arxiv.org/abs/2506.04381
PDF
-
Prompt Candidates, then Distill: A Teacher-Student Framework for LLM-driven Data Annotation
2025-06-04 11:42:37Mingxuan Xia, Haobo Wang, Yixuan Li, Zewei Yu, Jindong Wang, Junbo Zhao, Runze WuAbstract
Recently, Large Language Models (LLMs) have demonstrated significant potential for data annotation, markedly reducing the labor costs associated with downstream applications. However, existing methods mostly adopt an aggressive strategy by prompting LLM to determine a single gold label for each unlabeled sample. Due to the inherent uncertainty within LLMs, they often produce incorrect labels for difficult samples, severely compromising the data quality for downstream applications. Motivated by ambiguity aversion in human behaviors, we propose a novel candidate annotation paradigm wherein large language models are encouraged to output all possible labels when incurring uncertainty. To ensure unique labels are provided for downstream tasks, we develop a teacher-student framework CanDist that distills candidate annotations with a Small Language Model (SLM). We further provide a rigorous justification demonstrating that distilling candidate annotations from the teacher LLM offers superior theoretical guarantees compared to directly using single annotations. Extensive experiments across six text classification tasks validate the effectiveness of our proposed method. The source code is available at this https URL.
Abstract (translated)
最近，大型语言模型（LLMs）在数据标注方面展示了巨大的潜力，显著降低了下游应用的劳动力成本。然而，现有方法大多采用激进策略，通过提示LLM为每个未标记样本确定单一的“黄金标签”。由于LLM固有的不确定性，它们经常会为困难样本生成错误的标签，严重损害了下游应用程序的数据质量。 受到人类行为中规避模糊性的启发，我们提出了一种新颖的候选标注范式，在遇到不确定情况时鼓励大型语言模型输出所有可能的标签。为了确保为下游任务提供唯一的标签，我们开发了一个教师-学生框架CanDist，利用小型语言模型（SLM）对候选标注进行蒸馏。进一步地，我们提供了严格的理论依据，证明从老师LLM中蒸馏出候选注解相比直接使用单一注解具有更好的理论保证。 在六种文本分类任务上进行的广泛实验验证了我们提出的方法的有效性。源代码可在以下链接获得：[此处插入具体URL]。
URL
https://arxiv.org/abs/2506.03857
PDF
-
mdok of KInIT: Robustly Fine-tuned LLM for Binary and Multiclass AI-Generated Text Detection
2025-06-02 14:07:32Dominik MackoAbstract
The large language models (LLMs) are able to generate high-quality texts in multiple languages. Such texts are often not recognizable by humans as generated, and therefore present a potential of LLMs for misuse (e.g., plagiarism, spams, disinformation spreading). An automated detection is able to assist humans to indicate the machine-generated texts; however, its robustness to out-of-distribution data is still challenging. This notebook describes our mdok approach in robust detection, based on fine-tuning smaller LLMs for text classification. It is applied to both subtasks of Voight-Kampff Generative AI Detection 2025, providing remarkable performance in binary detection as well as in multiclass (1st rank) classification of various cases of human-AI collaboration.
Abstract (translated)
大型语言模型（LLMs）能够生成多语言的高质量文本。这些文本通常难以被人类识别为机器生成，因此它们存在被滥用的风险（例如抄袭、垃圾信息和虚假信息传播）。自动检测可以辅助人类识别机器生成的文本；然而，它在处理分布外数据时仍然具有挑战性。本笔记本描述了我们基于微调小型LLMs进行文本分类的方法mdok，在鲁棒性检测方面的应用。这种方法被应用于Voight-Kampff 2025生成AI检测竞赛的两个子任务中，并在二元检测和多类（第一名）分类的各种人类-AI合作案例中表现出显著性能。
URL
https://arxiv.org/abs/2506.01702
PDF
-
Domain Lexical Knowledge-based Word Embedding Learning for Text Classification under Small Data
2025-06-02 12:59:41Zixiao Zhu, Kezhi MaoAbstract
Pre-trained language models such as BERT have been proved to be powerful in many natural language processing tasks. But in some text classification applications such as emotion recognition and sentiment analysis, BERT may not lead to satisfactory performance. This often happens in applications where keywords play critical roles in the prediction of class labels. Our investigation found that the root cause of the problem is that the context-based BERT embedding of the keywords may not be discriminative enough to produce discriminative text representation for classification. Motivated by this finding, we develop a method to enhance word embeddings using domain-specific lexical knowledge. The knowledge-based embedding enhancement model projects the BERT embedding into a new space where within-class similarity and between-class difference are maximized. To implement the knowledge-based word embedding enhancement model, we also develop a knowledge acquisition algorithm for automatically collecting lexical knowledge from online open sources. Experiment results on three classification tasks, including sentiment analysis, emotion recognition and question answering, have shown the effectiveness of our proposed word embedding enhancing model. The codes and datasets are in this https URL.
Abstract (translated)
预训练语言模型（如BERT）已被证明在许多自然语言处理任务中非常强大。但在某些文本分类应用，例如情感识别和情感分析中，BERT可能无法达到令人满意的效果。这种情况往往发生在关键词在类别预测中起关键作用的应用场景中。我们的研究发现该问题的根本原因是基于上下文的BERT对关键词的嵌入不够具有区分性，从而不能生成用于分类的有效文本表示。 鉴于此发现，我们开发了一种使用特定领域的词汇知识来增强词嵌入的方法。这种基于知识的嵌入增强模型将BERT嵌入投影到一个新的空间，在该空间中同一类别的相似性和不同类别之间的差异被最大化了。为了实现基于知识的单词嵌入增强模型，我们也开发了一个自动从在线公开资源收集词汇知识的知识获取算法。 我们在情感分析、情绪识别和问答三项分类任务上的实验结果表明我们提出的词嵌入增强模型的有效性。代码和数据集可在以下链接中找到：[这里插入URL]。
URL
https://arxiv.org/abs/2506.01621
PDF
-
Efficient Ensemble for Fine-tuning Language Models on Multiple Datasets
2025-05-28 03:27:08Dongyue Li, Ziniu Zhang, Lu Wang, Hongyang R. ZhangAbstract
This paper develops an ensemble method for fine-tuning a language model to multiple datasets. Existing methods, such as quantized LoRA (QLoRA), are efficient when adapting to a single dataset. When training on multiple datasets of different tasks, a common setup in practice, it remains unclear how to design an efficient adaptation for fine-tuning language models. We propose to use an ensemble of multiple smaller adapters instead of a single adapter per task. We design an efficient algorithm that partitions $n$ datasets into $m$ groups, where $m$ is typically much smaller than $n$ in practice, and train one adapter for each group before taking a weighted combination to form the ensemble. The algorithm leverages a first-order approximation property of low-rank adaptation to quickly obtain the fine-tuning performances of dataset combinations since methods like LoRA stay close to the base model. Hence, we use the gradients of the base model to estimate its behavior during fine-tuning. Empirically, this approximation holds with less than $1\%$ error on models with up to $34$ billion parameters, leading to an estimation of true fine-tuning performances under $5\%$ error while speeding up computation compared to base fine-tuning by $105$ times. When applied to fine-tune Llama and GPT models on ten text classification tasks, our approach provides up to $10\%$ higher average test accuracy over QLoRA, with only $9\%$ more FLOPs. On a Llama model with $34$ billion parameters, an ensemble of QLoRA increases test accuracy by $3\%$ compared to QLoRA, with only $8\%$ more FLOPs.
Abstract (translated)
本文提出了一种针对多数据集微调语言模型的集成方法。现有技术如量化低秩适应（QLoRA）在单一数据集上的应用效率很高，但在面对不同任务的多个数据集进行训练时，如何设计高效的微调策略仍不清楚。我们建议使用多个较小适配器的集合而不是为每个任务单独创建一个适配器。具体来说，我们的方法将 $n$ 个数据集划分为 $m$ 组（其中 $m$ 在实践中通常远小于 $n$），然后为每一组训练一个适配器，并通过加权组合形成集成模型。 该算法利用了低秩适应的一阶近似性质，可以通过基础模型的梯度快速估计不同数据集组合在微调后的性能。这是因为像LoRA这样的方法在整个微调过程中仍保持接近基础模型的状态。因此，在计算上较为昂贵的实际微调之前，我们使用基础模型的梯度来估算其行为。 实验表明，这种近似误差低于1%，适用于高达340亿参数规模的语言模型，并且能够以约5%的误差范围内准确估计真正的微调性能，同时将基础模型微调的时间缩短了105倍。在对Llama和GPT模型进行十项文本分类任务的微调时，我们的方法相比QLoRA可提供高达10%的平均测试精度提升，并且仅增加了9%的浮点运算次数（FLOPs）。对于一个包含340亿参数的Llama模型而言，在使用QLoRA的基础上采用集合方法可以将测试准确度提高3%，同时只增加了8%的FLOPs。
URL
https://arxiv.org/abs/2505.21930
PDF
-
Leveraging large language models and traditional machine learning ensembles for ADHD detection from narrative transcripts
2025-05-27 15:22:01Yuxin Zhu, Yuting Guo, Noah Marchuck, Abeed Sarker, Yun WangAbstract
Despite rapid advances in large language models (LLMs), their integration with traditional supervised machine learning (ML) techniques that have proven applicability to medical data remains underexplored. This is particularly true for psychiatric applications, where narrative data often exhibit nuanced linguistic and contextual complexity, and can benefit from the combination of multiple models with differing characteristics. In this study, we introduce an ensemble framework for automatically classifying Attention-Deficit/Hyperactivity Disorder (ADHD) diagnosis (binary) using narrative transcripts. Our approach integrates three complementary models: LLaMA3, an open-source LLM that captures long-range semantic structure; RoBERTa, a pre-trained transformer model fine-tuned on labeled clinical narratives; and a Support Vector Machine (SVM) classifier trained using TF-IDF-based lexical features. These models are aggregated through a majority voting mechanism to enhance predictive robustness. The dataset includes 441 instances, including 352 for training and 89 for validation. Empirical results show that the ensemble outperforms individual models, achieving an F$_1$ score of 0.71 (95\% CI: [0.60-0.80]). Compared to the best-performing individual model (SVM), the ensemble improved recall while maintaining competitive precision. This indicates the strong sensitivity of the ensemble in identifying ADHD-related linguistic cues. These findings demonstrate the promise of hybrid architectures that leverage the semantic richness of LLMs alongside the interpretability and pattern recognition capabilities of traditional supervised ML, offering a new direction for robust and generalizable psychiatric text classification.
Abstract (translated)
尽管在大型语言模型（LLMs）方面取得了快速进展，但它们与传统监督式机器学习（ML）技术的结合仍未被充分探索。这些传统方法已在医疗数据的应用中证明了其有效性，尤其是在精神科应用领域，其中叙述性数据常常表现出复杂的语义和上下文特征，并且可以从多个具有不同特性的模型组合中受益。在这项研究中，我们介绍了一种用于自动分类注意力缺陷多动障碍（ADHD）诊断（二元分类）的叙事转录文本的集成框架。我们的方法整合了三种互补性模型：LLaMA3，一个开源的大规模语言模型，能够捕捉长程语义结构；RoBERTa，一种在标记临床叙述上进行了微调的预训练变换器模型；以及支持向量机（SVM）分类器，使用基于TF-IDF的词汇特征进行训练。这些模型通过多数投票机制整合起来以增强预测稳健性。该数据集包括441个实例，其中352个用于训练，89个用于验证。实证结果表明，集成方法优于单一模型的表现，达到了0.71的F1分数（95%置信区间：[0.60-0.80]）。与表现最佳的单个模型（SVM）相比，集成方法提高了召回率同时保持了竞争力的精度。这说明该集成系统在识别ADHD相关语言线索方面具有较强的敏感性。这些发现展示了混合架构的潜力，即利用LLMs的语义丰富性和传统监督式机器学习的可解释性和模式识别能力，在稳健和泛化的精神科文本分类上开辟了一条新的方向。
URL
https://arxiv.org/abs/2505.21324
PDF
-
Information-Theoretic Complementary Prompts for Improved Continual Text Classification
2025-05-27 09:22:14Duzhen Zhang, Yong Ren, Chenxing Li, Dong Yu, Tielin ZhangAbstract
Continual Text Classification (CTC) aims to continuously classify new text data over time while minimizing catastrophic forgetting of previously acquired knowledge. However, existing methods often focus on task-specific knowledge, overlooking the importance of shared, task-agnostic knowledge. Inspired by the complementary learning systems theory, which posits that humans learn continually through the interaction of two systems -- the hippocampus, responsible for forming distinct representations of specific experiences, and the neocortex, which extracts more general and transferable representations from past experiences -- we introduce Information-Theoretic Complementary Prompts (InfoComp), a novel approach for CTC. InfoComp explicitly learns two distinct prompt spaces: P(rivate)-Prompt and S(hared)-Prompt. These respectively encode task-specific and task-invariant knowledge, enabling models to sequentially learn classification tasks without relying on data replay. To promote more informative prompt learning, InfoComp uses an information-theoretic framework that maximizes mutual information between different parameters (or encoded representations). Within this framework, we design two novel loss functions: (1) to strengthen the accumulation of task-specific knowledge in P-Prompt, effectively mitigating catastrophic forgetting, and (2) to enhance the retention of task-invariant knowledge in S-Prompt, improving forward knowledge transfer. Extensive experiments on diverse CTC benchmarks show that our approach outperforms previous state-of-the-art methods.
Abstract (translated)
连续文本分类（CTC）的目标是在时间的推移中持续对新文本数据进行分类，同时尽量减少之前获得的知识的灾难性遗忘。然而，现有的方法往往专注于特定任务的知识，而忽视了共享、无任务特异性知识的重要性。受互补学习系统理论的启发——该理论认为人类通过两个系统的相互作用来实现连续学习：海马体负责形成特定经历的独特表示，新皮层则从过去的经验中提取更为普遍和可转移的表示——我们提出了信息论互补提示（InfoComp），这是一种新颖的CTC方法。InfoComp明确地学习两种不同的提示空间：P(rivate)-Prompt（私有提示）和S(hared)-Prompt（共享提示）。它们分别编码特定任务的知识和无任务特异性的知识，使模型能够在不依赖数据重播的情况下逐步学习分类任务。 为了促进更具信息量的提示学习，InfoComp采用了一个基于信息论的框架，该框架最大化了不同参数或表示之间的互信息。在此框架内，我们设计了两个新的损失函数：（1）用于增强P-Prompt中特定任务知识的积累，有效减轻灾难性遗忘；（2）用于提高S-Prompt中无任务特异性知识的保留度，从而改进向前的知识传递。 在多种CTC基准上的广泛实验表明，我们的方法优于先前最先进的方法。
URL
https://arxiv.org/abs/2505.20933
PDF
-
Large Language Models in the Task of Automatic Validation of Text Classifier Predictions
2025-05-24 13:19:03Aleksandr TsymbalovAbstract
Machine learning models for text classification are trained to predict a class for a given text. To do this, training and validation samples must be prepared: a set of texts is collected, and each text is assigned a class. These classes are usually assigned by human annotators with different expertise levels, depending on the specific classification task. Collecting such samples from scratch is labor-intensive because it requires finding specialists and compensating them for their work; moreover, the number of available specialists is limited, and their productivity is constrained by human factors. While it may not be too resource-intensive to collect samples once, the ongoing need to retrain models (especially in incremental learning pipelines) to address data drift (also called model drift) makes the data collection process crucial and costly over the model's entire lifecycle. This paper proposes several approaches to replace human annotators with Large Language Models (LLMs) to test classifier predictions for correctness, helping ensure model quality and support high-quality incremental learning.
Abstract (translated)
用于文本分类的机器学习模型是被训练来为给定的文本预测一个类别。为了实现这一点，必须准备训练和验证样本：收集一组文本，并且每个文本都由人工标注员分配一个类别。这些类通常是根据特定的分类任务的不同要求，由具有不同专业知识水平的人工标注员进行标注。从头开始收集这样的样本是非常费力的工作，因为它需要找到专家并支付他们的费用；此外，可用的专家数量有限，而且他们的工作效率受到人类因素的影响而受限。 虽然一次性收集一些样本可能不会消耗太多资源，但在整个模型生命周期中持续地为了应对数据漂移（也称为模型漂移）重新训练模型（特别是在增量学习管道中），使数据收集过程变得至关重要且成本高昂。本文提出了一些方法来用大型语言模型（LLMs）替代人工标注员以测试分类器预测的准确性，这有助于确保模型的质量，并支持高质量的增量学习。
URL
https://arxiv.org/abs/2505.18688
PDF
-
Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change
2025-05-24 11:45:46Murathan Kurfal{\i}, Shorouq Zahra, Joakim Nivre, Gabriele MessoriAbstract
Climate-Eval is a comprehensive benchmark designed to evaluate natural language processing models across a broad range of tasks related to climate change. Climate-Eval aggregates existing datasets along with a newly developed news classification dataset, created specifically for this release. This results in a benchmark of 25 tasks based on 13 datasets, covering key aspects of climate discourse, including text classification, question answering, and information extraction. Our benchmark provides a standardized evaluation suite for systematically assessing the performance of large language models (LLMs) on these tasks. Additionally, we conduct an extensive evaluation of open-source LLMs (ranging from 2B to 70B parameters) in both zero-shot and few-shot settings, analyzing their strengths and limitations in the domain of climate change.
Abstract (translated)
Climate-Eval 是一个全面的基准测试，旨在评估与气候变化相关的广泛任务中的自然语言处理模型。该基准整合了现有的数据集以及为此次发布特别开发的一套新闻分类数据集。这产生了基于13个数据集的25项任务，涵盖了气候讨论的关键方面，包括文本分类、问答和信息提取。我们的基准测试提供了一个标准化的评估工具包，用于系统地评估大型语言模型（LLMs）在这些任务上的表现。此外，我们在零样本和少样本设置中对开源LLM（从20亿参数到700亿参数）进行了广泛的评估，分析了它们在气候变化领域的优缺点。
URL
https://arxiv.org/abs/2505.18653
PDF
-
Less Context, Same Performance: A RAG Framework for Resource-Efficient LLM-Based Clinical NLP
2025-05-23 16:13:08Satya Narayana Cheetirala, Ganesh Raut, Dhavalkumar Patel, Fabio Sanatana, Robert Freeman, Matthew A Levin, Girish N. Nadkarni, Omar Dawkins, Reba Miller, Randolph M. Steinhagen, Eyal Klang, Prem TimsinaAbstract
Long text classification is challenging for Large Language Models (LLMs) due to token limits and high computational costs. This study explores whether a Retrieval Augmented Generation (RAG) approach using only the most relevant text segments can match the performance of processing entire clinical notes with large context LLMs. We begin by splitting clinical documents into smaller chunks, converting them into vector embeddings, and storing these in a FAISS index. We then retrieve the top 4,000 words most pertinent to the classification query and feed these consolidated segments into an LLM. We evaluated three LLMs (GPT4o, LLaMA, and Mistral) on a surgical complication identification task. Metrics such as AUC ROC, precision, recall, and F1 showed no statistically significant differences between the RAG based approach and whole-text processing (p > 0.05p > 0.05). These findings indicate that RAG can significantly reduce token usage without sacrificing classification accuracy, providing a scalable and cost effective solution for analyzing lengthy clinical documents.
Abstract (translated)
长文本分类对于大型语言模型（LLMs）来说是一个挑战，原因在于令牌限制和高昂的计算成本。这项研究探讨了一种使用检索增强生成（RAG）方法仅利用最相关的文本片段是否能够与全量临床记录的大上下文LLMs处理相媲美。我们首先将临床文档分割成更小的部分，将其转换为向量嵌入，并存储在FAISS索引中。然后，我们将从索引中检索出4000个最相关词汇的段落，并将其输入到LLM模型进行处理。我们在三种不同的LLMs（GPT-4、LLaMA和Mistral）上对一个外科并发症识别任务进行了评估。在AUC ROC、精确度、召回率以及F1分数等指标下，基于RAG的方法与全量文本处理之间没有统计显著差异（p > 0.05）。这些结果表明，RAG可以在不牺牲分类准确性的情况下大幅减少令牌使用，并提供了一种可扩展且成本效益高的解决方案来分析较长的临床文档。
URL
https://arxiv.org/abs/2505.20320
PDF
-
Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs?
2025-05-23 05:46:42Junyan Zhang, Yiming Huang, Shuliang Liu, Yubo Gao, Xuming HuAbstract
The rapid adoption of LLMs has overshadowed the potential advantages of traditional BERT-like models in text classification. This study challenges the prevailing "LLM-centric" trend by systematically comparing three category methods, i.e., BERT-like models fine-tuning, LLM internal state utilization, and zero-shot inference across six high-difficulty datasets. Our findings reveal that BERT-like models often outperform LLMs. We further categorize datasets into three types, perform PCA and probing experiments, and identify task-specific model strengths: BERT-like models excel in pattern-driven tasks, while LLMs dominate those requiring deep semantics or world knowledge. Based on this, we propose TaMAS, a fine-grained task selection strategy, advocating for a nuanced, task-driven approach over a one-size-fits-all reliance on LLMs.
Abstract (translated)
大型语言模型（LLMs）的迅速采用已经掩盖了传统BERT类模型在文本分类中的潜在优势。本研究通过系统地比较三种类别方法——即对BERT类模型进行微调、利用LLM内部状态以及零样本推理，在六个高难度数据集上挑战现有的“LLM中心论”趋势。我们的发现表明，传统的BERT类模型往往优于大型语言模型（LLMs）。我们进一步将数据集分类为三类，并进行了PCA和探测实验，以识别任务特定的模型优势：BERT类模型在基于模式的任务中表现出色，而大型语言模型则在需要深度语义或世界知识的任务中占据主导地位。基于此，我们提出了TaMAS，一种细粒度的任务选择策略，提倡采取精细、任务驱动的方法而非一刀切地依赖LLMs。
URL
https://arxiv.org/abs/2505.18215
PDF
-
Small Language Models in the Real World: Insights from Industrial Text Classification
2025-05-21 23:39:24Lujun Li, Lama Sleem, Niccolo' Gentile, Geoffrey Nichil, Radu StateAbstract
With the emergence of ChatGPT, Transformer models have significantly advanced text classification and related tasks. Decoder-only models such as Llama exhibit strong performance and flexibility, yet they suffer from inefficiency on inference due to token-by-token generation, and their effectiveness in text classification tasks heavily depends on prompt quality. Moreover, their substantial GPU resource requirements often limit widespread adoption. Thus, the question of whether smaller language models are capable of effectively handling text classification tasks emerges as a topic of significant interest. However, the selection of appropriate models and methodologies remains largely underexplored. In this paper, we conduct a comprehensive evaluation of prompt engineering and supervised fine-tuning methods for transformer-based text classification. Specifically, we focus on practical industrial scenarios, including email classification, legal document categorization, and the classification of extremely long academic texts. We examine the strengths and limitations of smaller models, with particular attention to both their performance and their efficiency in Video Random-Access Memory (VRAM) utilization, thereby providing valuable insights for the local deployment and application of compact models in industrial settings.
Abstract (translated)
随着ChatGPT的出现，Transformer模型在文本分类及相关任务上取得了显著的进步。例如Llama这样的解码器自回归模型展现出强大的性能和灵活性，但由于逐令牌生成过程中的效率低下，在推理阶段表现不佳，并且它们在文本分类任务上的效果严重依赖于提示的质量。此外，这些模型对GPU资源的巨大需求往往限制了其广泛应用。因此，关于较小规模的语言模型是否能够有效处理文本分类任务的问题引起了广泛关注。然而，选择合适的模型和方法仍然是一个未被充分探索的领域。 本文中，我们全面评估了基于Transformer的文本分类中的提示工程和监督微调方法。具体来说，我们的研究集中在实际工业场景上，包括电子邮件分类、法律文档归类以及对极其长篇学术文章进行分类的任务。我们将重点关注小型模型的优势与局限性，并特别关注它们在性能及视频随机访问存储器（VRAM）利用率方面的表现，从而为紧凑型模型在工业环境中的本地部署和应用提供宝贵的见解。
URL
https://arxiv.org/abs/2505.16078
PDF
-
Large Language Models as Computable Approximations to Solomonoff Induction
2025-05-21 17:35:08Jun Wan, Lingrui MeiAbstract
The rapid advancement of large language models (LLMs) calls for a rigorous theoretical framework to explain their empirical success. While significant progress has been made in understanding LLM behaviors, existing theoretical frameworks remain fragmented in explaining emergent phenomena through a unified mathematical lens. We establish the first formal connection between LLM architectures and Algorithmic Information Theory (AIT) by proving two fundamental results: (1) the training process computationally approximates Solomonoff prior through loss minimization interpreted as program length optimization, and (2) next-token prediction implements approximate Solomonoff induction. We leverage AIT to provide a unified theoretical explanation for in-context learning, few-shot learning, and scaling laws. Furthermore, our theoretical insights lead to a principled method for few-shot example selection that prioritizes samples where models exhibit lower predictive confidence. We demonstrate through experiments on diverse text classification benchmarks that this strategy yields significant performance improvements, particularly for smaller model architectures, when compared to selecting high-confidence examples. Our framework bridges the gap between theoretical foundations and practical LLM behaviors, providing both explanatory power and actionable insights for future model development.
Abstract (translated)
大型语言模型（LLM）的快速发展需要一个严格的理论框架来解释其实验成功的原因。尽管在理解LLM行为方面取得了显著进展，但现有的理论框架仍然缺乏统一的数学视角来解释新兴现象。我们首次建立了LLM架构与算法信息论（AIT）之间的正式联系，并证明了两个基本结果：(1) 通过损失最小化实现程序长度优化的过程计算上近似于Solomonoff先验；(2) 下一令牌预测实现了对Solomonoff归纳的近似实施。我们利用AIT提供了一个统一的理论解释，涵盖了上下文学习、少量样本学习以及规模法则。此外，我们的理论见解为少样本示例选择提供了基于原则的方法，优先考虑模型表现出较低预测置信度的情况下的样本。通过在多种文本分类基准上的实验表明，与选取高置信度示例相比，该策略能显著提高较小架构模型的性能。我们的框架弥合了理论基础和实际LLM行为之间的差距，为未来模型的发展提供了解释力和实用性的洞察。
URL
https://arxiv.org/abs/2505.15784
PDF
-
Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications
2025-05-20 21:12:58Fadel M. Megahed, Ying-Ju Chen, L. Allision Jones-Farmer, Younghwa Lee, Jiawei Brooke Wang, Inez M. ZwetslootAbstract
This study introduces a framework for evaluating consistency in large language model (LLM) binary text classification, addressing the lack of established reliability assessment methods. Adapting psychometric principles, we determine sample size requirements, develop metrics for invalid responses, and evaluate intra- and inter-rater reliability. Our case study examines financial news sentiment classification across 14 LLMs (including claude-3-7-sonnet, gpt-4o, deepseek-r1, gemma3, llama3.2, phi4, and command-r-plus), with five replicates per model on 1,350 articles. Models demonstrated high intra-rater consistency, achieving perfect agreement on 90-98% of examples, with minimal differences between expensive and economical models from the same families. When validated against StockNewsAPI labels, models achieved strong performance (accuracy 0.76-0.88), with smaller models like gemma3:1B, llama3.2:3B, and claude-3-5-haiku outperforming larger counterparts. All models performed at chance when predicting actual market movements, indicating task constraints rather than model limitations. Our framework provides systematic guidance for LLM selection, sample size planning, and reliability assessment, enabling organizations to optimize resources for classification tasks.
Abstract (translated)
这项研究提出了一种评估大型语言模型（LLM）二元文本分类一致性的框架，旨在解决现有可靠性评估方法不足的问题。通过借鉴心理测量学原理，我们确定了样本量的需求，开发了无效响应的度量标准，并对评分者内和评分者间的可靠性进行了评价。我们的案例研究考察了14个大型语言模型（包括claude-3-7-sonnet、gpt-4o、deepseek-r1、gemma3、llama3.2、phi4和command-r-plus）在财务新闻情绪分类方面的表现，每个模型重复进行五次测试，共分析了1,350篇文章。各模型展示了较高的评分者内一致性，在90%-98%的样本中实现了完全一致的意见，并且来自同一家族的昂贵型和经济型模型之间差异很小。当与StockNewsAPI标签验证时，这些模型表现出色（准确率在0.76到0.88之间），其中较小规模的模型如gemma3:1B、llama3.2:3B以及claude-3-5-haiku的表现优于其大型同类。所有模型在预测实际市场走势时表现均接近随机水平，这表明任务难度而非模型能力限制了性能。我们的框架为LLM选择、样本量规划和可靠性评估提供了系统性指导，帮助组织优化资源以应对分类任务的挑战。
URL
https://arxiv.org/abs/2505.14918
PDF
-
KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approaches
2025-05-20 18:00:01Mingquan Feng, Yixin Huang, Yifan Fu, Shaobo Wang, Junchi YanAbstract
The design of optimization algorithms for neural networks remains a critical challenge, with most existing methods relying on heuristic adaptations of gradient-based approaches. This paper introduces KO (Kinetics-inspired Optimizer), a novel neural optimizer inspired by kinetic theory and partial differential equation (PDE) simulations. We reimagine the training dynamics of network parameters as the evolution of a particle system governed by kinetic principles, where parameter updates are simulated via a numerical scheme for the Boltzmann transport equation (BTE) that models stochastic particle collisions. This physics-driven approach inherently promotes parameter diversity during optimization, mitigating the phenomenon of parameter condensation, i.e. collapse of network parameters into low-dimensional subspaces, through mechanisms analogous to thermal diffusion in physical systems. We analyze this property, establishing both a mathematical proof and a physical interpretation. Extensive experiments on image classification (CIFAR-10/100, ImageNet) and text classification (IMDB, Snips) tasks demonstrate that KO consistently outperforms baseline optimizers (e.g., Adam, SGD), achieving accuracy improvements while computation cost remains comparable.
Abstract (translated)
神经网络的优化算法设计仍然是一个关键挑战，大多数现有的方法依赖于基于梯度的方法的启发式改进。本文介绍了KO（Kinetics-inspired Optimizer），这是一种受动力学理论和偏微分方程（PDE）模拟启发的新颖神经网络优化器。我们重新构思了网络参数训练的动力学，将其视为由动力学原理支配的粒子系统的演化过程，并通过玻尔兹曼输运方程（BTE）的数值方案来模拟参数更新，该模型描述了随机粒子碰撞的过程。这种物理驱动的方法内在地促进了优化过程中参数的多样性，通过类似于物理系统中热扩散机制的方式缓解参数凝聚现象，即网络参数坍缩到低维子空间的现象。我们分析了这一特性，并建立了数学证明和物理解释。 在图像分类（CIFAR-10/100、ImageNet）和文本分类（IMDB、Snips）任务的广泛实验中表明，KO始终优于基线优化器（例如Adam、SGD），并在计算成本相当的情况下提高了准确性。
URL
https://arxiv.org/abs/2505.14777
PDF
