Text_Classification
-
Systematic Comparative Analysis of Large Pretrained Language Models on Contextualized Medication Event Extraction
2025-09-23 16:48:28Tariq Abdul-Quddoos, Xishuang Dong, Lijun QianAbstract
Attention-based models have become the leading approach in modeling medical language for Natural Language Processing (NLP) in clinical notes. These models outperform traditional techniques by effectively capturing contextual rep- resentations of language. In this research a comparative analysis is done amongst pre- trained attention based models namely Bert Base, BioBert, two variations of Bio+Clinical Bert, RoBerta, and Clinical Long- former on task related to Electronic Health Record (EHR) information extraction. The tasks from Track 1 of Harvard Medical School's 2022 National Clinical NLP Challenges (n2c2) are considered for this comparison, with the Contextualized Medication Event Dataset (CMED) given for these task. CMED is a dataset of unstructured EHRs and annotated notes that contain task relevant information about the EHRs. The goal of the challenge is to develop effective solutions for extracting contextual information related to patient medication events from EHRs using data driven methods. Each pre-trained model is fine-tuned and applied on CMED to perform medication extraction, medical event detection, and multi-dimensional medication event context classification. Pro- cessing methods are also detailed for breaking down EHRs for compatibility with the applied models. Performance analysis has been carried out using a script based on constructing medical terms from the evaluation portion of CMED with metrics including recall, precision, and F1-Score. The results demonstrate that models pre-trained on clinical data are more effective in detecting medication and medication events, but Bert Base, pre- trained on general domain data showed to be the most effective for classifying the context of events related to medications.
Abstract (translated)
基于注意力的模型已经成为自然语言处理（NLP）在临床笔记中建模医学语言的领先方法。这些模型通过有效捕捉语言的上下文表示，超越了传统的技术。在这项研究中，对几个预训练的注意力模型进行比较分析，包括Bert Base、BioBert、两种不同的Bio+Clinical Bert变体、RoBerta和Clinical LongFormer，在与电子健康记录（EHR）信息提取相关的任务上进行了对比。这些任务来自哈佛医学院2022年全国临床NLP挑战赛（n2c2）的Track 1，使用的数据集是上下文化的药物事件数据集（CMED），该数据集包含未结构化和标注良好的EHR注释，其中包含了与EHR相关的任务相关信息。比赛的目标是开发有效的方法从EHR中提取患者用药情况的相关信息，并采用基于数据驱动的方法。 每个预训练模型都在CMED上进行了微调并应用于药物提取、医疗事件检测以及多维度的药物事件上下文分类。还详细说明了处理方法，用于分解EHR以使其与应用模型兼容。性能分析使用了一个基于从CMED评估部分构建医学术语的脚本进行，该脚本采用了包括召回率（recall）、精确度（precision）和F1分数在内的指标。 结果显示，在检测药物及药物事件方面，经过临床数据预训练的模型更有效；然而，在分类与药物相关的事件上下文时，通用领域数据上预训练的Bert Base被证明是最有效的。
URL
https://arxiv.org/abs/2509.19224
PDF
-
Enhancing Cross-Lingual Transfer through Reversible Transliteration: A Huffman-Based Approach for Low-Resource Languages
2025-09-22 08:24:26Wenhao Zhuang, Yuan Sun, Xiaobing ZhaoAbstract
As large language models (LLMs) are trained on increasingly diverse and extensive multilingual corpora, they demonstrate cross-lingual transfer capabilities. However, these capabilities often fail to effectively extend to low-resource languages, particularly those utilizing non-Latin scripts. While transliterating low-resource languages into Latin script presents a natural solution, there currently lacks a comprehensive framework for integrating transliteration into LLMs training and deployment. Taking a pragmatic approach, this paper innovatively combines character transliteration with Huffman coding to design a complete transliteration framework. Our proposed framework offers the following advantages: 1) Compression: Reduces storage requirements for low-resource language content, achieving up to 50% reduction in file size and 50-80% reduction in token count. 2) Accuracy: Guarantees 100% lossless conversion from transliterated text back to the source language. 3) Efficiency: Eliminates the need for vocabulary expansion for low-resource languages, improving training and inference efficiency. 4) Scalability: The framework can be extended to other low-resource languages. We validate the effectiveness of our framework across multiple downstream tasks, including text classification, machine reading comprehension, and machine translation. Experimental results demonstrate that our method significantly enhances the model's capability to process low-resource languages while maintaining performance on high-resource languages. Our data and code are publicly available at this https URL.
Abstract (translated)
随着大型语言模型（LLMs）在日益多样化和广泛的多语种语料库上进行训练，它们展示出了跨语言迁移能力。然而，这些能力往往未能有效地扩展到低资源语言中，尤其是那些使用非拉丁字母的语言。虽然将低资源语言转写为拉丁字母提供了一个自然的解决方案，但目前尚缺乏一个全面的框架来将这种转写集成到LLMs的训练和部署过程中。本文采用了一种实用的方法，创新性地结合了字符转写与霍夫曼编码，设计出一套完整的转写框架。 我们提出的框架具有以下优势： 1) 压缩：减少了低资源语言内容所需的存储要求，文件大小可减少高达50%，标记数量可以减少50-80%。 2) 准确性：确保从转写字体文本转换回源语言时无损。 3) 效率：消除了对低资源语言词汇表扩展的需求，提高了训练和推理的效率。 4) 可扩展性：该框架可以应用于其他低资源语言。 我们在多个下游任务中验证了我们框架的有效性，包括文本分类、机器阅读理解以及机器翻译。实验结果表明，我们的方法显著提升了模型处理低资源语言的能力，并且同时保持在高资源语言上的性能不变。我们的数据和代码可在以下网址公开获取：[此URL]（请根据实际情况替换为具体的链接）。
URL
https://arxiv.org/abs/2509.17493
PDF
-
GTA: Supervised-Guided Reinforcement Learning for Text Classification with Large Language Models
2025-09-15 16:33:56Min Zeng, Jinfei Sun, Xueyou Luo, Caiquan Liu, Shiqi Zhang, Li Xie, Xiaoxin ChenAbstract
In natural language processing tasks, pure reinforcement learning (RL) fine-tuning methods often suffer from inefficient exploration and slow convergence; while supervised fine-tuning (SFT) methods, although efficient in training, have limited performance ceiling and less solid theoretical foundation compared to RL. To address efficiency-capability trade-off, we propose the Guess-Think-Answer (GTA) framework that combines the efficiency of SFT with the capability gains of RL in a unified training paradigm. GTA works by having the model first produce a provisional guess (optimized via cross-entropy loss), then reflect on this guess before generating the final answer, with RL rewards shaping both the final output and the format of the entire GTA structure. This hybrid approach achieves both faster convergence than pure RL and higher performance ceiling than pure SFT. To mitigate gradient conflicts between the two training signals, we employ loss masking and gradient constraints. Empirical results on four text classification benchmarks demonstrate that GTA substantially accelerates convergence while outperforming both standalone SFT and RL baselines.
Abstract (translated)
在自然语言处理任务中，纯粹的强化学习（RL）微调方法常常因为探索效率低下和收敛速度慢而面临挑战；而监督微调（SFT）方法虽然训练效率高，但其性能上限较低，并且与RL相比理论基础较弱。为了解决效率与能力之间的权衡问题，我们提出了Guess-Think-Answer (GTA) 框架，该框架结合了SFT的高效性和RL的能力提升，在统一的训练范式中实现了两者的优点。GTA的工作机制是让模型首先生成一个初步猜测（通过交叉熵损失进行优化），然后基于这个初步猜测进行反思，最终输出答案，其中强化学习奖励不仅影响最终输出，还塑造整个GTA结构的形式。这种混合方法比纯粹的RL收敛更快，并且在性能上限方面优于纯粹的SFT。为了缓解两种训练信号之间的梯度冲突，我们采用了损失掩码和梯度约束技术。实验证明，在四个文本分类基准数据集上，GTA不仅大大加速了收敛速度，还超过了独立使用的SFT和RL基线方法的表现。
URL
https://arxiv.org/abs/2509.12108
PDF
-
Beyond Token Limits: Assessing Language Model Performance on Long Text Classification
2025-09-12 12:47:28Mikl\'os Seb\H{o}k, Viktor Kov\'acs, Martin B\'an\'oczy, Daniel M{\o}ller Eriksen, Nathalie Neptune, Philippe RoussilleAbstract
The most widely used large language models in the social sciences (such as BERT, and its derivatives, e.g. RoBERTa) have a limitation on the input text length that they can process to produce predictions. This is a particularly pressing issue for some classification tasks, where the aim is to handle long input texts. One such area deals with laws and draft laws (bills), which can have a length of multiple hundred pages and, therefore, are not particularly amenable for processing with models that can only handle e.g. 512 tokens. In this paper, we show results from experiments covering 5 languages with XLM-RoBERTa, Longformer, GPT-3.5, GPT-4 models for the multiclass classification task of the Comparative Agendas Project, which has a codebook of 21 policy topic labels from education to health care. Results show no particular advantage for the Longformer model, pre-trained specifically for the purposes of handling long inputs. The comparison between the GPT variants and the best-performing open model yielded an edge for the latter. An analysis of class-level factors points to the importance of support and substance overlaps between specific categories when it comes to performance on long text inputs.
Abstract (translated)
在社会科学领域中广泛使用的大规模语言模型（如BERT及其衍生版本，例如RoBERTa）在处理生成预测的输入文本长度方面存在限制。对于需要处理长篇输入文本的一些分类任务而言，这个问题尤为突出。其中一个研究领域涉及法律和法案草案，这些文件可能长达数百页，因此不适合只能处理512个令牌等有限长度文本的模型进行处理。在这篇文章中，我们展示了使用XLM-RoBERTa、Longformer、GPT-3.5及GPT-4模型在比较议程项目（Comparative Agendas Project）多类分类任务中的实验结果，该任务涉及从教育到医疗保健等21个政策主题标签的代码簿。结果显示，专门针对处理长输入文本而预训练的Longformer模型并无显著优势。将GPT变体与表现最佳的开源模型进行比较后发现，后者具有明显的优势。对类别层面因素的分析表明，在处理长篇文本时，特定类别之间的支持和实质重叠对于性能至关重要。
URL
https://arxiv.org/abs/2509.10199
PDF
-
Are Humans as Brittle as Large Language Models?
2025-09-09 15:56:51Jiahui Li, Sean Papay, Roman KlingerAbstract
The output of large language models (LLM) is unstable, due to both non-determinism of the decoding process as well as to prompt brittleness. While the intrinsic non-determinism of LLM generation may mimic existing uncertainty in human annotations through distributional shifts in outputs, it is largely assumed, yet unexplored, that the prompt brittleness effect is unique to LLMs. This raises the question: do human annotators show similar sensitivity to instruction changes? If so, should prompt brittleness in LLMs be considered problematic? One may alternatively hypothesize that prompt brittleness correctly reflects human annotation variances. To fill this research gap, we systematically compare the effects of prompt modifications on LLMs and identical instruction modifications for human annotators, focusing on the question of whether humans are similarly sensitive to prompt perturbations. To study this, we prompt both humans and LLMs for a set of text classification tasks conditioned on prompt variations. Our findings indicate that both humans and LLMs exhibit increased brittleness in response to specific types of prompt modifications, particularly those involving the substitution of alternative label sets or label formats. However, the distribution of human judgments is less affected by typographical errors and reversed label order than that of LLMs.
Abstract (translated)
大型语言模型（LLM）的输出不稳定，这既由于解码过程中的非确定性，也由于提示的脆弱性。虽然LLM生成内在的非确定性可以通过输出分布的变化模拟现有标注中的不确定性，但人们普遍认为但尚未充分探讨的是，提示脆弱性的效果是特定于大型语言模型的独特现象。这就提出了一个问题：人类注释者对指令变化是否表现出类似的敏感度？如果是的话，那么大型语言模型中的提示脆性是否应被视为问题？另一种假设可能是，提示脆性准确反映了人类标注的变化。为填补这一研究空白，我们系统地比较了对LLM和人类注释者的相同指令修改的提示更改效果，并重点关注人类是否同样敏感于提示扰动的问题。为此，我们在一系列文本分类任务中分别向人类和大型语言模型提出了根据提示变化条件的任务。我们的发现表明，无论是人类还是大型语言模型，在面对特定类型的提示更改时都会表现出更大的脆弱性，尤其是那些涉及替代标签集或标签格式替换的情况。然而，与LLM相比，人类判断的分布受到拼写错误和标签顺序反转的影响较小。
URL
https://arxiv.org/abs/2509.07869
PDF
-
Hybrid Topic-Semantic Labeling and Graph Embeddings for Unsupervised Legal Document Clustering
2025-08-31 20:53:59Deepak Bastola, Woohyeok ChoiAbstract
Legal documents pose unique challenges for text classification due to their domain-specific language and often limited labeled data. This paper proposes a hybrid approach for classifying legal texts by combining unsupervised topic and graph embeddings with a supervised model. We employ Top2Vec to learn semantic document embeddings and automatically discover latent topics, and Node2Vec to capture structural relationships via a bipartite graph of legal documents. The embeddings are combined and clustered using KMeans, yielding coherent groupings of documents. Our computations on a legal document dataset demonstrate that the combined Top2Vec+Node2Vec approach improves clustering quality over text-only or graph-only embeddings. We conduct a sensitivity analysis of hyperparameters, such as the number of clusters and the dimensionality of the embeddings, and demonstrate that our method achieves competitive performance against baseline Latent Dirichlet Allocation (LDA) and Non-Negative Matrix Factorization (NMF) models. Key findings indicate that while the pipeline presents an innovative approach to unsupervised legal document analysis by combining semantic topic modeling with graph embedding techniques, its efficacy is contingent upon the quality of initial topic generation and the representational power of the chosen embedding models for specialized legal language. Strategic recommendations include the exploration of domain-specific embeddings, more comprehensive hyperparameter tuning for Node2Vec, dynamic determination of cluster numbers, and robust human-in-the-loop validation processes to enhance legal relevance and trustworthiness. The pipeline demonstrates potential for exploratory legal data analysis and as a precursor to supervised learning tasks but requires further refinement and domain-specific adaptation for practical legal applications.
Abstract (translated)
法律文件的分类对于文本分类来说具有独特的挑战，因为这些文档含有特定领域的语言，并且通常标注数据有限。本文提出了一种混合方法来对法律文本进行分类，该方法结合了无监督的主题和图嵌入与有监督模型。我们利用Top2Vec学习语义文档嵌入并自动发现潜在主题，同时使用Node2Vec通过法律文件的二部图捕捉结构关系。将这些嵌入合并并通过KMeans聚类分析生成一致的文档组群。我们在一个法律文档数据集上的计算表明，结合了Top2Vec和Node2Vec的方法在文本或仅基于图的嵌入方法上提高了聚类的质量。 我们对超参数（如聚类数量和嵌入维度）进行了敏感性分析，并证明我们的方法在基线潜在狄利克雷分配（LDA）和非负矩阵分解（NMF）模型中具有竞争力。关键发现表明，虽然此流程通过结合语义主题建模与图嵌入技术提供了创新的无监督法律文档分析方法，但其有效性取决于初始主题生成的质量以及所选嵌入模型对特定法律语言的表达能力。 战略建议包括探索领域特异性嵌入、更全面地调整Node2Vec中的超参数、动态确定聚类数量以及实施稳健的人机交互验证过程来增强法律相关性和可信度。该流程展示了在法律数据探索性分析和作为监督学习任务之前的前奏方面的潜力，但需要进一步的细化及特定领域的适应以满足实际应用需求。 以下是对原文的主要翻译： 法律文件分类因其领域专有语言及标注数据有限而具有独特挑战。本文提出了一种结合无监督主题与图嵌入以及监督模型对法律文本进行分类的方法。我们使用Top2Vec学习语义文档嵌入并自动发现潜在话题，通过在法律文档二部图中运用Node2Vec捕捉结构关系。我们将这些嵌入合并并通过KMeans聚类生成一致的文档组群。 我们在一个法律文件数据集上的计算表明，结合Top2Vec与Node2Vec的方法相较于仅使用文本或仅基于图的嵌入方法能提升聚类质量。我们对超参数（如聚类数量、嵌入维度等）进行了敏感性分析，并证明该方法在基线潜在狄利克雷分配（LDA）和非负矩阵分解（NMF）模型中具有竞争力。 关键发现表明，虽然此流程通过结合语义主题建模与图嵌入技术提供了创新的无监督法律文档分析方法，但其有效性依赖于初始话题生成的质量及所选嵌入模型对特定语言表达能力。战略建议包括探索领域特异性嵌入、更细致地调整Node2Vec中的超参数、动态确定聚类数量以及实施稳健的人机交互验证过程来增强相关性和可信度。 该流程展示了法律数据的探索性分析和作为监督学习任务前导方面的潜力，但需进一步细化及特定领域的适应以满足实际应用需求。
URL
https://arxiv.org/abs/2509.00990
PDF
-
Performance Analysis of Supervised Machine Learning Algorithms for Text Classification
2025-08-31 20:30:52Sadia Zaman Mishu, S M RafiuddinAbstract
The demand for text classification is growing significantly in web searching, data mining, web ranking, recommendation systems, and so many other fields of information and technology. This paper illustrates the text classification process on different datasets using some standard supervised machine learning techniques. Text documents can be classified through various kinds of classifiers. Labeled text documents are used to classify the text in supervised classifications. This paper applies these classifiers on different kinds of labeled documents and measures the accuracy of the classifiers. An Artificial Neural Network (ANN) model using Back Propagation Network (BPN) is used with several other models to create an independent platform for labeled and supervised text classification process. An existing benchmark approach is used to analyze the performance of classification using labeled documents. Experimental analysis on real data reveals which model works well in terms of classification accuracy.
Abstract (translated)
文本分类的需求在网页搜索、数据挖掘、网页排名、推荐系统以及其他许多信息和技术领域中正显著增长。本文通过使用一些标准的监督机器学习技术，在不同的数据集上展示了文本分类的过程。可以通过多种类型的分类器对文本文件进行分类。在监督分类中，带有标签的文本文档被用来分类文本。本文将这些分类器应用于不同种类的标记文档，并衡量这些分类器的准确性。 使用人工神经网络（ANN）模型和反向传播网络（BPN），与其他几种模型一起构建了一个独立平台，用于对带标签和监督下的文本分类过程进行处理。利用现有的基准方法来分析使用标记文档时分类性能的表现。通过真实数据上的实验分析揭示了哪种模型在分类准确性方面表现良好。
URL
https://arxiv.org/abs/2509.00983
PDF
-
Selective Retrieval-Augmentation for Long-Tail Legal Text Classification
Abstract
Legal text classification is a fundamental NLP task in the legal domain. Benchmark datasets in this area often exhibit a long-tail label distribution, where many labels are underrepresented, leading to poor model performance on rare classes. This paper proposes Selective Retrieval-Augmentation (SRA) as a solution to this problem. SRA focuses on augmenting samples belonging to low-frequency labels in the training set, preventing the introduction of noise for well-represented classes, and requires no changes to the model architecture. Retrieval is performed only from the training data to ensure there is no potential information leakage, removing the need for external corpora simultaneously. The proposed SRA method is tested on two legal text classification benchmark datasets with long-tail distributions: LEDGAR (single-label) and UNFAIR-ToS (multi-label). The results indicate that SRA attains higher micro-F1 and macro-F1 scores compared to all current LexGLUE baselines across both datasets, illustrating consistent improvements in long-tail legal text classification. The code repository is available at: this https URL
Abstract (translated)
法律文本分类是法律领域中的基本自然语言处理任务。在这个领域的基准数据集中，标签分布往往呈现出长尾特性，即许多标签的代表性不足，导致模型在罕见类别上的表现较差。本文提出了一种名为选择性检索增强（Selective Retrieval-Augmentation，简称SRA）的方法来解决这个问题。SRA专注于增强训练集中的低频标签样本，同时避免为那些已有足够代表性的类引入噪声，并且无需对模型架构进行任何更改。检索仅从训练数据中进行，以确保没有潜在的信息泄露风险，并且不需要外部语料库。 本文提出的SRA方法在两个具有长尾分布的法律文本分类基准数据集上进行了测试：LEDGAR（单标签）和UNFAIR-ToS（多标签）。实验结果显示，在这两个数据集中，与所有当前LexGLUE基线相比，SRA获得了更高的微平均F1分数和宏平均F1分数，表明其在长尾法律文本分类方面取得了持续的改进。该方法的代码库可在此处获得：[此链接](https://this https URL)。 请注意，原文中的“this https URL”应更正为实际的有效链接地址以提供访问相关代码资源的可能性。
URL
https://arxiv.org/abs/2508.19997
PDF
-
Dhati+: Fine-tuned Large Language Models for Arabic Subjectivity Evaluation
2025-08-27 15:20:12Slimane Bellaouar, Attia Nehar, Soumia Souffi, Mounia BouameurAbstract
Despite its significance, Arabic, a linguistically rich and morphologically complex language, faces the challenge of being under-resourced. The scarcity of large annotated datasets hampers the development of accurate tools for subjectivity analysis in Arabic. Recent advances in deep learning and Transformers have proven highly effective for text classification in English and French. This paper proposes a new approach for subjectivity assessment in Arabic textual data. To address the dearth of specialized annotated datasets, we developed a comprehensive dataset, AraDhati+, by leveraging existing Arabic datasets and collections (ASTD, LABR, HARD, and SANAD). Subsequently, we fine-tuned state-of-the-art Arabic language models (XLM-RoBERTa, AraBERT, and ArabianGPT) on AraDhati+ for effective subjectivity classification. Furthermore, we experimented with an ensemble decision approach to harness the strengths of individual models. Our approach achieves a remarkable accuracy of 97.79\,\% for Arabic subjectivity classification. Results demonstrate the effectiveness of the proposed approach in addressing the challenges posed by limited resources in Arabic language processing.
Abstract (translated)
尽管阿拉伯语作为一种语言丰富且形态复杂的语言具有重要性，但它面临着资源不足的挑战。大型标注数据集的匮乏阻碍了开发准确进行主观分析工具的发展。最近在深度学习和Transformer方面的进展已经证明对英语和法语文本分类非常有效。本文提出了一种新的方法来评估阿拉伯语文本数据中的主观性。为了应对专门标注数据集缺乏的问题，我们通过利用现有的阿拉伯语数据集和集合（ASTD、LABR、HARD 和 SANAD）开发了一个全面的数据集AraDhati+。随后，我们将最先进的阿拉伯语言模型（XLM-RoBERTa、AraBERT 和 ArabianGPT）在AraDhati+上进行了微调，以实现有效的主观性分类。此外，我们还尝试了集成决策方法来利用各个模型的优势。我们的方法实现了97.79% 的阿拉伯语主观性分类准确率。结果表明，所提出的方法在解决阿拉伯语文本处理中有限资源带来的挑战方面非常有效。
URL
https://arxiv.org/abs/2508.19966
PDF
-
Yet Unnoticed in LSTM: Binary Tree Based Input Reordering, Weight Regularization, and Gate Nonlinearization
2025-08-27 10:40:02Mojtaba MoattariAbstract
LSTM models used in current Machine Learning literature and applications, has a promising solution for permitting long term information using gating mechanisms that forget and reduce effect of current input information. However, even with this pipeline, they do not optimally focus on specific old index or long-term information. This paper elaborates upon input reordering approaches to prioritize certain input indices. Moreover, no LSTM based approach is found in the literature that examines weight normalization while choosing the right weight and exponent of Lp norms through main supervised loss function. In this paper, we find out which norm best finds relationship between weights to either smooth or sparsify them. Lastly, gates, as weighted representations of inputs and states, which control reduction-extent of current input versus previous inputs (~ state), are not nonlinearized enough (through a small FFNN). As analogous to attention mechanisms, gates easily filter current information to bold (emphasize on) past inputs. Nonlinearized gates can more easily tune up to peculiar nonlinearities of specific input in the past. This type of nonlinearization is not proposed in the literature, to the best of author's knowledge. The proposed approaches are implemented and compared with a simple LSTM to understand their performance in text classification tasks. The results show they improve accuracy of LSTM.
Abstract (translated)
当前的机器学习文献和应用中使用的LSTM模型通过使用门控机制来遗忘旧信息并减少当前输入的影响，从而允许可长期存储信息。然而，即使有了这些机制，它们也无法在处理特定旧索引或长时间信息时达到最佳聚焦效果。本文探讨了重新排序输入的方法以优先考虑某些输入索引。 此外，在现有的文献中尚未发现任何基于LSTM的方法会考察在选择合适的权重和$L_p$范数指数时进行权重归一化的做法，特别是在主要的监督损失函数范围内。本文通过研究哪种范数能最好地找到权重之间的关系（无论是平滑还是稀疏化），来寻找最优解决方案。 另外，门控机制作为输入和状态的加权表示，控制了当前输入与先前输入（~ 状态）之间的影响程度。但这些门控机制并未进行足够的非线性处理（通过小型前馈神经网络）。类似注意力机制，在过滤当前信息以突出显示过去的信息时，它们很容易做到这一点。经过非线性的门控可以更有效地调整个别过去的输入的特殊非线性。 据作者所知，这种类型的非线性化在文献中尚未被提出过。本文提出的几种方法与简单的LSTM进行对比测试，用以评估其在文本分类任务中的性能表现。实验结果表明，这些方法确实提高了LSTM模型的准确率。
URL
https://arxiv.org/abs/2509.00087
PDF
-
Automatic Prompt Optimization with Prompt Distillation
2025-08-26 12:46:58Viktor N. Zhuravlev, Artur R. Khairullin, Ernest A. Dyagin, Alena N. Sitkina, Nikita I. KulinAbstract
Autoprompting is the process of automatically selecting optimized prompts for language models, which is gaining popularity due to the rapid development of prompt engineering driven by extensive research in the field of large language models (LLMs). This paper presents DistillPrompt -- a novel autoprompting method based on large language models that employs a multi-stage integration of task-specific information into prompts using training data. DistillPrompt utilizes distillation, compression, and aggregation operations to explore the prompt space more thoroughly. The method was tested on different datasets for text classification and generation tasks using the t-lite-instruct-0.1 language model. The results demonstrate a significant average improvement (e.g., 20.12% across the entire dataset compared to Grips) in key metrics over existing methods in the field, establishing DistillPrompt as one of the most effective non-gradient approaches in autoprompting.
Abstract (translated)
自动生成提示词（Autoprompting）是指自动选择优化后的提示词供语言模型使用，这一方法由于在大规模语言模型（LLMs）领域中广泛研究而推动的提示工程快速发展变得越来越受欢迎。本文介绍了一种基于大型语言模型的新颖自动生成提示词方法——DistillPrompt。该方法通过多阶段整合任务特定信息到提示词中，并利用训练数据进行操作，包括蒸馏、压缩和聚合等步骤来更彻底地探索提示空间。 在文本分类和生成任务的不同数据集上使用t-lite-instruct-0.1语言模型对DistillPrompt进行了测试。实验结果显示，在关键性能指标方面，与现有方法相比（例如，相较于Grips在整个数据集上的平均改进达20.12%），DistillPrompt表现出显著的平均改善效果，并确立了其作为非梯度自动生成提示词方法中最有效的技术之一的地位。
URL
https://arxiv.org/abs/2508.18992
PDF
-
Efficient Zero-Shot Long Document Classification by Reducing Context Through Sentence Ranking
2025-08-24 18:52:37Prathamesh Kokate, Mitali Sarnaik, Manavi Khopade, Mukta Takalikar, Raviraj JoshiAbstract
Transformer-based models like BERT excel at short text classification but struggle with long document classification (LDC) due to input length limitations and computational inefficiencies. In this work, we propose an efficient, zero-shot approach to LDC that leverages sentence ranking to reduce input context without altering the model architecture. Our method enables the adaptation of models trained on short texts, such as headlines, to long-form documents by selecting the most informative sentences using a TF-IDF-based ranking strategy. Using the MahaNews dataset of long Marathi news articles, we evaluate three context reduction strategies that prioritize essential content while preserving classification accuracy. Our results show that retaining only the top 50\% ranked sentences maintains performance comparable to full-document inference while reducing inference time by up to 35\%. This demonstrates that sentence ranking is a simple yet effective technique for scalable and efficient zero-shot LDC.
Abstract (translated)
基于Transformer的模型（如BERT）在处理短文本分类时表现出色，但在长文档分类（LDC）方面却面临输入长度限制和计算效率低下的挑战。为此，我们提出了一种高效且无需微调的方法来应对长文档分类问题，该方法利用句子排名减少输入上下文而不改变模型架构。我们的方法允许将训练于短文本（如新闻标题）上的模型应用于长篇文档，并通过使用基于TF-IDF的排名策略选择最具信息量的句子实现这一目标。 我们使用MahaNews数据集中的长马拉地语新闻文章，评估了三种减少上下文的方法，这些方法优先考虑关键内容的同时保持分类准确性。实验结果表明，保留前50%排名最高的句子可以将推理时间最多缩短35%，而同时维持与完整文档推理时相当的性能表现。 这表明基于句子排名的技术是一种简单且有效的方法，可用于零样本长文档分类（LDC），以实现可扩展性和效率。
URL
https://arxiv.org/abs/2508.17490
PDF
-
GRAID: Synthetic Data Generation with Geometric Constraints and Multi-Agentic Reflection for Harmful Content Detection
2025-08-23 15:09:58Melissa Kazemi Rad, Alberto Purpura, Himanshu Kumar, Emily Chen, Mohammad Shahed SorowerAbstract
We address the problem of data scarcity in harmful text classification for guardrailing applications and introduce GRAID (Geometric and Reflective AI-Driven Data Augmentation), a novel pipeline that leverages Large Language Models (LLMs) for dataset augmentation. GRAID consists of two stages: (i) generation of geometrically controlled examples using a constrained LLM, and (ii) augmentation through a multi-agentic reflective process that promotes stylistic diversity and uncovers edge cases. This combination enables both reliable coverage of the input space and nuanced exploration of harmful content. Using two benchmark data sets, we demonstrate that augmenting a harmful text classification dataset with GRAID leads to significant improvements in downstream guardrail model performance.
Abstract (translated)
我们解决了护拦应用中危害性文本分类数据不足的问题，并引入了GRAID（几何和反思型AI驱动的数据增强），这是一种利用大型语言模型（LLMs）进行数据集扩增的新型流程。GRAID包括两个阶段：(i) 使用受约束的LLM生成几何控制示例，以及(ii) 通过一个多代理反思过程进行扩充，该过程促进了风格多样性并揭示了边缘案例。这种组合使得输入空间的可靠覆盖和有害内容的细腻探索成为可能。使用两个基准数据集，我们证明将GRAID用于扩增危害性文本分类数据集可以显著提高下游护拦模型的表现。
URL
https://arxiv.org/abs/2508.17057
PDF
-
LLM-as-classifier: Semi-Supervised, Iterative Framework for Hierarchical Text Classification using Large Language Models
2025-08-22 15:47:17Doohee You, Andy Parisi, Zach Vander Velden, Lara Dantas InojosaAbstract
The advent of Large Language Models (LLMs) has provided unprecedented capabilities for analyzing unstructured text data. However, deploying these models as reliable, robust, and scalable classifiers in production environments presents significant methodological challenges. Standard fine-tuning approaches can be resource-intensive and often struggle with the dynamic nature of real-world data distributions, which is common in the industry. In this paper, we propose a comprehensive, semi-supervised framework that leverages the zero- and few-shot capabilities of LLMs for building hierarchical text classifiers as a framework for a solution to these industry-wide challenges. Our methodology emphasizes an iterative, human-in-the-loop process that begins with domain knowledge elicitation and progresses through prompt refinement, hierarchical expansion, and multi-faceted validation. We introduce techniques for assessing and mitigating sequence-based biases and outline a protocol for continuous monitoring and adaptation. This framework is designed to bridge the gap between the raw power of LLMs and the practical need for accurate, interpretable, and maintainable classification systems in industry applications.
Abstract (translated)
大型语言模型（LLMs）的出现为分析非结构化文本数据提供了前所未有的能力。然而，在生产环境中部署这些模型作为可靠、稳健且可扩展的分类器面临着重大的方法论挑战。传统的微调方法通常资源消耗大，且难以应对现实世界数据分布的动态性，这是业界常见的问题。本文提出了一种全面的半监督框架，利用LLMs的零样本和小样本能力来构建分层文本分类器，以解决这些行业普遍面临的难题。我们的方法强调了一个迭代的人机交互流程，该流程从领域知识提取开始，经过提示优化、层次扩展及多维度验证等阶段逐步推进。我们引入了评估和缓解序列偏差的技术，并制定了持续监控和适应的协议。此框架旨在弥合LLMs原始能力与行业中准确、可解释且易于维护分类系统实际需求之间的差距。
URL
https://arxiv.org/abs/2508.16478
PDF
-
Strategic Sample Selection for Improved Clean-Label Backdoor Attacks in Text Classification
2025-08-21 19:53:26Onur Alp Kirci, M. Emre GursoyAbstract
Backdoor attacks pose a significant threat to the integrity of text classification models used in natural language processing. While several dirty-label attacks that achieve high attack success rates (ASR) have been proposed, clean-label attacks are inherently more difficult. In this paper, we propose three sample selection strategies to improve attack effectiveness in clean-label scenarios: Minimum, Above50, and Below50. Our strategies identify those samples which the model predicts incorrectly or with low confidence, and by injecting backdoor triggers into such samples, we aim to induce a stronger association between the trigger patterns and the attacker-desired target label. We apply our methods to clean-label variants of four canonical backdoor attacks (InsertSent, WordInj, StyleBkd, SynBkd) and evaluate them on three datasets (IMDB, SST2, HateSpeech) and four model types (LSTM, BERT, DistilBERT, RoBERTa). Results show that the proposed strategies, particularly the Minimum strategy, significantly improve the ASR over random sample selection with little or no degradation in the model's clean accuracy. Furthermore, clean-label attacks enhanced by our strategies outperform BITE, a state of the art clean-label attack method, in many configurations.
Abstract (translated)
后门攻击对自然语言处理中使用的文本分类模型的完整性构成了重大威胁。虽然已经提出了几种能够实现高攻击成功率（ASR）的脏标签攻击，但干净标签攻击本质上更加困难。在本文中，我们提出三种样本选择策略以提高干净标签场景中的攻击效果：Minimum、Above50 和 Below50。我们的策略识别模型预测错误或置信度较低的样本，并通过向这些样本注入后门触发器，旨在增强触发模式与攻击者希望的目标标签之间的关联性。我们将这四种经典的后门攻击（InsertSent、WordInj、StyleBkd、SynBkd）的干净标签变体应用于三个数据集（IMDB、SST2、HateSpeech），并对四种模型类型（LSTM、BERT、DistilBERT、RoBERTa）进行了评估。结果表明，所提出的策略特别是Minimum 策略，在提高 ASR 方面显著优于随机样本选择，并且对模型的干净准确率几乎没有或没有退化影响。此外，在许多配置中，通过我们的策略增强的干净标签攻击超越了最先进的干净标签攻击方法BITE。
URL
https://arxiv.org/abs/2508.15934
PDF
-
Comparing energy consumption and accuracy in text classification inference
2025-08-19 18:00:08Johannes Zschache, Tilman HartwigAbstract
The increasing deployment of large language models (LLMs) in natural language processing (NLP) tasks raises concerns about energy efficiency and sustainability. While prior research has largely focused on energy consumption during model training, the inference phase has received comparatively less attention. This study systematically evaluates the trade-offs between model accuracy and energy consumption in text classification inference across various model architectures and hardware configurations. Our empirical analysis shows that the best-performing model in terms of accuracy can also be energy-efficient, while larger LLMs tend to consume significantly more energy with lower classification accuracy. We observe substantial variability in inference energy consumption ($<$mWh to $>$kWh), influenced by model type, model size, and hardware specifications. Additionally, we find a strong correlation between inference energy consumption and model runtime, indicating that execution time can serve as a practical proxy for energy usage in settings where direct measurement is not feasible. These findings have implications for sustainable AI development, providing actionable insights for researchers, industry practitioners, and policymakers seeking to balance performance and resource efficiency in NLP applications.
Abstract (translated)
在自然语言处理（NLP）任务中部署大规模语言模型（LLMs）引发了关于能源效率和可持续性的担忧。尽管先前的研究主要集中在模型训练过程中的能耗问题，但在推理阶段的关注相对较少。本研究系统地评估了不同模型架构和硬件配置下文本分类推理过程中模型准确性和能量消耗之间的权衡关系。我们的实证分析表明，在准确性方面表现最佳的模型也可以是能源效率高的，而更大的LLMs通常会消耗更多能量且分类精度更低。 我们观察到推理阶段的能量消耗存在显著差异（从不到毫瓦时到超过千瓦时），这受到模型类型、模型大小以及硬件规格的影响。此外，我们发现推理能耗与模型运行时间之间有很强的相关性，表明在无法直接测量的情况下，执行时间可以作为能源使用的一个实用替代指标。 这些研究结果对可持续AI的发展具有重要影响，为寻求平衡性能和资源效率的研究人员、行业从业者及政策制定者提供了可操作的见解。
URL
https://arxiv.org/abs/2508.14170
PDF
-
Compressed Models are NOT Trust-equivalent to Their Large Counterparts
2025-08-19 05:49:39Rohit Raj Rai, Chirag Kothari, Siddhesh Shelke, Amit AwekarAbstract
Large Deep Learning models are often compressed before being deployed in a resource-constrained environment. Can we trust the prediction of compressed models just as we trust the prediction of the original large model? Existing work has keenly studied the effect of compression on accuracy and related performance measures. However, performance parity does not guarantee trust-equivalence. We propose a two-dimensional framework for trust-equivalence evaluation. First, interpretability alignment measures whether the models base their predictions on the same input features. We use LIME and SHAP tests to measure the interpretability alignment. Second, calibration similarity measures whether the models exhibit comparable reliability in their predicted probabilities. It is assessed via ECE, MCE, Brier Score, and reliability diagrams. We conducted experiments using BERT-base as the large model and its multiple compressed variants. We focused on two text classification tasks: natural language inference and paraphrase identification. Our results reveal low interpretability alignment and significant mismatch in calibration similarity. It happens even when the accuracies are nearly identical between models. These findings show that compressed models are not trust-equivalent to their large counterparts. Deploying compressed models as a drop-in replacement for large models requires careful assessment, going beyond performance parity.
Abstract (translated)
大型深度学习模型在部署到资源受限的环境中之前通常会被压缩。我们可以像信任原始大规模模型那样完全信赖被压缩后的模型吗？现有研究已经深入探讨了压缩对准确性和相关性能指标的影响，但性能上的等价并不保证信任度的一致性。我们提出了一个用于评估信任度一致性的二维框架。 首先，可解释性一致性衡量的是这些模型是否基于相同的输入特征进行预测。我们使用LIME（局部可解释的模型无关方法）和SHAP（分值的重要性和属性贡献）测试来测量这一方面。 其次，校准相似性衡量的是这些模型在预测概率中的可靠性是否相当一致。通过预期校准误差（ECE）、最大校准误差（MCE）、布赖尔得分以及可靠性的图表来进行评估。 我们使用BERT-base作为大型原始模型，并对其多个压缩版本进行了实验。我们重点关注了两个文本分类任务：自然语言推理和同义句识别。我们的结果显示，即使在准确率几乎相同的情况下，可解释性一致性也很低且校准相似性存在显著差异。这表明被压缩的模型与它们的大规模原型并不具备等价的信任度。 因此，在将压缩模型作为大规模模型的直接替代品进行部署时，需要超越性能等价的考量来进行仔细评估。
URL
https://arxiv.org/abs/2508.13533
PDF
-
A BERT-based Hierarchical Classification Model with Applications in Chinese Commodity Classification
2025-08-13 16:10:47Kun Liu, Tuozhen Liu, Feifei Wang, Rui PanAbstract
Existing e-commerce platforms heavily rely on manual annotation for product categorization, which is inefficient and inconsistent. These platforms often employ a hierarchical structure for categorizing products; however, few studies have leveraged this hierarchical information for classification. Furthermore, studies that consider hierarchical information fail to account for similarities and differences across various hierarchical categories. Herein, we introduce a large-scale hierarchical dataset collected from the JD e-commerce platform (this http URL), comprising 1,011,450 products with titles and a three-level category structure. By making this dataset openly accessible, we provide a valuable resource for researchers and practitioners to advance research and applications associated with product categorization. Moreover, we propose a novel hierarchical text classification approach based on the widely used Bidirectional Encoder Representations from Transformers (BERT), called Hierarchical Fine-tuning BERT (HFT-BERT). HFT-BERT leverages the remarkable text feature extraction capabilities of BERT, achieving prediction performance comparable to those of existing methods on short texts. Notably, our HFT-BERT model demonstrates exceptional performance in categorizing longer short texts, such as books.
Abstract (translated)
现有的电子商务平台在商品分类时严重依赖于人工标注，这不仅效率低下而且一致性差。这些平台通常采用分层结构对产品进行分类；然而，鲜有研究利用这种层级信息来进行分类。更重要的是，考虑到层级信息的研究未能充分考虑不同层级类别之间的相似性和差异性。 在此背景下，我们引入了一个大规模的分层数据集，该数据集从京东电子商务平台收集而来（此链接），包含1,011,450个具有标题和三级类别的产品。通过公开提供这个数据集，我们为研究者和从业者提供了推进与商品分类相关的研究和应用的重要资源。 此外，我们提出了一种基于广泛使用的双向编码表示从变压器(BERT)的新型分层文本分类方法，称为层次微调BERT（HFT-BERT）。HFT-BERT利用了BERT卓越的文本特征提取能力，在短文本上的预测性能可与现有方法媲美。值得注意的是，我们的HFT-BERT模型在对较长的短文本进行分类时表现出色，例如书籍类商品。 这个研究为电子商务领域的商品自动分类提供了新的视角和工具，并且通过提供一个大规模、结构化的数据集以及一个改进的分类框架，有望提高整个行业的效率和准确性。
URL
https://arxiv.org/abs/2508.15800
PDF
-
SinLlama - A Large Language Model for Sinhala
2025-08-12 17:49:34H. W. K. Aravinda, Rashad Sirajudeen, Samith Karunathilake, Nisansa de Silva, Surangika Ranathunga, Rishemjit KaurAbstract
Low-resource languages such as Sinhala are often overlooked by open-source Large Language Models (LLMs). In this research, we extend an existing multilingual LLM (Llama-3-8B) to better serve Sinhala. We enhance the LLM tokenizer with Sinhala specific vocabulary and perform continual pre-training on a cleaned 10 million Sinhala corpus, resulting in the SinLlama model. This is the very first decoder-based open-source LLM with explicit Sinhala support. When SinLlama was instruction fine-tuned for three text classification tasks, it outperformed base and instruct variants of Llama-3-8B by a significant margin.
Abstract (translated)
资源较少的语言，如僧伽罗语，在开源大型语言模型（LLM）中往往被忽视。在本研究中，我们将现有的多语言LLM（Llama-3-8B）扩展，以更好地服务于僧伽罗语。我们通过增加针对僧伽罗语的特定词汇来增强LLM分词器，并使用经过清洗的1000万条僧伽罗语文本进行了持续预训练，从而生成了SinLlama模型。这是首个具有明确僧伽罗语支持的基于解码开源大型语言模型。在对三个文本分类任务进行指令微调时，SinLlama的表现远远超过了基础版本和指令变体的Llama-3-8B模型。
URL
https://arxiv.org/abs/2508.09115
PDF
-
Data-Efficient Biomedical In-Context Learning: A Diversity-Enhanced Submodular Perspective
2025-08-11 16:13:21Jun Wang, Zaifu Zhan, Qixin Zhang, Mingquan Lin, Meijia Song, Rui ZhangAbstract
Recent progress in large language models (LLMs) has leveraged their in-context learning (ICL) abilities to enable quick adaptation to unseen biomedical NLP tasks. By incorporating only a few input-output examples into prompts, LLMs can rapidly perform these new tasks. While the impact of these demonstrations on LLM performance has been extensively studied, most existing approaches prioritize representativeness over diversity when selecting examples from large corpora. To address this gap, we propose Dual-Div, a diversity-enhanced data-efficient framework for demonstration selection in biomedical ICL. Dual-Div employs a two-stage retrieval and ranking process: First, it identifies a limited set of candidate examples from a corpus by optimizing both representativeness and diversity (with optional annotation for unlabeled data). Second, it ranks these candidates against test queries to select the most relevant and non-redundant demonstrations. Evaluated on three biomedical NLP tasks (named entity recognition (NER), relation extraction (RE), and text classification (TC)) using LLaMA 3.1 and Qwen 2.5 for inference, along with three retrievers (BGE-Large, BMRetriever, MedCPT), Dual-Div consistently outperforms baselines-achieving up to 5% higher macro-F1 scores-while demonstrating robustness to prompt permutations and class imbalance. Our findings establish that diversity in initial retrieval is more critical than ranking-stage optimization, and limiting demonstrations to 3-5 examples maximizes performance efficiency.
Abstract (translated)
最近在大型语言模型（LLMs）方面的进展，利用了它们的上下文学习（ICL）能力来快速适应未见过的生物医学自然语言处理任务。通过将少量输入输出示例整合到提示中，LLMs能够迅速执行这些新任务。尽管已经对这些示范如何影响LLM性能进行了广泛研究，但现有方法在从大型语料库中选择示例时通常优先考虑代表性而非多样性。为了解决这一空白，我们提出了Dual-Div，这是一种增强多样性的高效数据框架，用于生物医学ICL中的示范选择。Dual-Div采用两阶段的检索和排序过程：首先，通过优化代表性和多样性（可选地针对未标记数据进行注释），它从语料库中识别出一组有限的候选示例；其次，在测试查询的基础上对这些候选者进行排名，以选择最相关且非冗余的示范。在三项生物医学NLP任务（命名实体识别(NER)、关系抽取(RE)和文本分类(TC)）上进行了评估，使用LLaMA 3.1 和 Qwen 2.5 进行推理，并结合了三种检索器（BGE-Large、BMRetriever、MedCPT），Dual-Div始终超越基线方法，在所有任务中宏F1分数提高了高达5%，同时展示了对提示排列和类别不平衡的鲁棒性。我们的研究结果表明，初始检索阶段中的多样性比排序阶段优化更为关键，并且将示范限制为3-5个示例可以最大化性能效率。
URL
https://arxiv.org/abs/2508.08140
PDF
