Text_Classification
-
Additive Large Language Models for Semi-Structured Text
2025-11-14 23:06:16Karthikeyan K, Raghuveer Thirukovalluru, David CarlsonAbstract
Large Language Models have advanced clinical text classification, but their opaque predictions remain a critical barrier to practical adoption in research and clinical settings where investigators and physicians need to understand which parts of a patient's record drive risk signals. To address this challenge, we introduce \textbf{CALM}, short for \textbf{Classification with Additive Large Language Models}, an interpretable framework for semi-structured text where inputs are composed of semantically meaningful components, such as sections of an admission note or question-answer fields from an intake form. CALM predicts outcomes as the additive sum of each component's contribution, making these contributions part of the forward computation itself and enabling faithful explanations at both the patient and population level. The additive structure also enables clear visualizations, such as component-level risk curves similar to those used in generalized additive models, making the learned relationships easier to inspect and communicate. Although CALM expects semi-structured inputs, many clinical documents already have this form, and similar structure can often be automatically extracted from free-text notes. CALM achieves performance comparable to conventional LLM classifiers while improving trust, supporting quality-assurance checks, and revealing clinically meaningful patterns during model development and auditing.
Abstract (translated)
URL
https://arxiv.org/abs/2511.11922
PDF
-
Ellipsoid-Based Decision Boundaries for Open Intent Classification
2025-11-13 11:31:14Yuetian Zou, Hanlei Zhang, Hua Xu, Songze Li, Long XiaoAbstract
Textual open intent classification is crucial for real-world dialogue systems, enabling robust detection of unknown user intents without prior knowledge and contributing to the robustness of the system. While adaptive decision boundary methods have shown great potential by eliminating manual threshold tuning, existing approaches assume isotropic distributions of known classes, restricting boundaries to balls and overlooking distributional variance along different directions. To address this limitation, we propose EliDecide, a novel method that learns ellipsoid decision boundaries with varying scales along different feature directions. First, we employ supervised contrastive learning to obtain a discriminative feature space for known samples. Second, we apply learnable matrices to parameterize ellipsoids as the boundaries of each known class, offering greater flexibility than spherical boundaries defined solely by centers and radii. Third, we optimize the boundaries via a novelly designed dual loss function that balances empirical and open-space risks: expanding boundaries to cover known samples while contracting them against synthesized pseudo-open samples. Our method achieves state-of-the-art performance on multiple text intent benchmarks and further on a question classification dataset. The flexibility of the ellipsoids demonstrates superior open intent detection capability and strong potential for generalization to more text classification tasks in diverse complex open-world scenarios.
Abstract (translated)
URL
https://arxiv.org/abs/2511.16685
PDF
-
Fractional neural attention for efficient multiscale sequence processing
2025-11-13 11:27:39Cheng Kevin Qu, Andrew Ly, Pulin GongAbstract
Attention mechanisms underpin the computational power of Transformer models, which have achieved remarkable success across diverse domains. Yet understanding and extending the principles underlying self-attention remains a key challenge for advancing artificial intelligence. Drawing inspiration from the multiscale dynamics of biological attention and from dynamical systems theory, we introduce Fractional Neural Attention (FNA), a principled, neuroscience-inspired framework for multiscale information processing. FNA models token interactions through Lévy diffusion governed by the fractional Laplacian, intrinsically realizing simultaneous short- and long-range dependencies across multiple scales. This mechanism yields greater expressivity and faster information mixing, advancing the foundational capacity of Transformers. Theoretically, we show that FNA's dynamics are governed by the fractional diffusion equation, and that the resulting attention networks exhibit larger spectral gaps and shorter path lengths -- mechanistic signatures of enhanced computational efficiency. Empirically, FNA achieves competitive text-classification performance even with a single layer and a single head; it also improves performance in image processing and neural machine translation. Finally, the diffusion map algorithm from geometric harmonics enables dimensionality reduction of FNA weights while preserving the intrinsic structure of embeddings and hidden states. Together, these results establish FNA as a principled mechanism connecting self-attention, stochastic dynamics, and geometry, providing an interpretable, biologically grounded foundation for powerful, neuroscience-inspired AI.
Abstract (translated)
URL
https://arxiv.org/abs/2511.10208
PDF
-
State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting?
2025-11-11 08:54:15Taja Kuzman Punger\v{s}ek, Peter Rupnik, Ivan Porupski, Vuk Dini\'c, Nikola Ljube\v{s}i\'cAbstract
Until recently, fine-tuned BERT-like models provided state-of-the-art performance on text classification tasks. With the rise of instruction-tuned decoder-only models, commonly known as large language models (LLMs), the field has increasingly moved toward zero-shot and few-shot prompting. However, the performance of LLMs on text classification, particularly on less-resourced languages, remains under-explored. In this paper, we evaluate the performance of current language models on text classification tasks across several South Slavic languages. We compare openly available fine-tuned BERT-like models with a selection of open-source and closed-source LLMs across three tasks in three domains: sentiment classification in parliamentary speeches, topic classification in news articles and parliamentary speeches, and genre identification in web texts. Our results show that LLMs demonstrate strong zero-shot performance, often matching or surpassing fine-tuned BERT-like models. Moreover, when used in a zero-shot setup, LLMs perform comparably in South Slavic languages and English. However, we also point out key drawbacks of LLMs, including less predictable outputs, significantly slower inference, and higher computational costs. Due to these limitations, fine-tuned BERT-like models remain a more practical choice for large-scale automatic text annotation.
Abstract (translated)
URL
https://arxiv.org/abs/2511.07989
PDF
-
Breaking the Adversarial Robustness-Performance Trade-off in Text Classification via Manifold Purification
2025-11-11 06:39:01Chenhao Dang, Jing MaAbstract
A persistent challenge in text classification (TC) is that enhancing model robustness against adversarial attacks typically degrades performance on clean data. We argue that this challenge can be resolved by modeling the distribution of clean samples in the encoder embedding manifold. To this end, we propose the Manifold-Correcting Causal Flow (MC^2F), a two-module system that operates directly on sentence embeddings. A Stratified Riemannian Continuous Normalizing Flow (SR-CNF) learns the density of the clean data manifold. It identifies out-of-distribution embeddings, which are then corrected by a Geodesic Purification Solver. This solver projects adversarial points back onto the learned manifold via the shortest path, restoring a clean, semantically coherent representation. We conducted extensive evaluations on text classification (TC) across three datasets and multiple adversarial attacks. The results demonstrate that our method, MC^2F, not only establishes a new state-of-the-art in adversarial robustness but also fully preserves performance on clean data, even yielding modest gains in accuracy.
Abstract (translated)
URL
https://arxiv.org/abs/2511.07888
PDF
-
Learn to Select: Exploring Label Distribution Divergence for In-Context Demonstration Selection in Text Classification
2025-11-10 08:04:14Ye Jiang, Taihang Wang, Youzheng Liu, Yimin Wang, Yuhan Xia, Yunfei LongAbstract
In-context learning (ICL) for text classification, which uses a few input-label demonstrations to describe a task, has demonstrated impressive performance on large language models (LLMs). However, the selection of in-context demonstrations plays a crucial role and can significantly affect LLMs' performance. Most existing demonstration selection methods primarily focus on semantic similarity between test inputs and demonstrations, often overlooking the importance of label distribution alignment. To address this limitation, we propose a two-stage demonstration selection method, TopK + Label Distribution Divergence (L2D), which leverages a fine-tuned BERT-like small language model (SLM) to generate label distributions and calculate their divergence for both test inputs and candidate demonstrations. This enables the selection of demonstrations that are not only semantically similar but also aligned in label distribution with the test input. Extensive experiments across seven text classification benchmarks show that our method consistently outperforms previous demonstration selection strategies. Further analysis reveals a positive correlation between the performance of LLMs and the accuracy of the underlying SLMs used for label distribution estimation.
Abstract (translated)
URL
https://arxiv.org/abs/2511.10675
PDF
-
Implicit Federated In-context Learning For Task-Specific LLM Fine-Tuning
2025-11-10 06:34:29Dongcheng Li, Junhan Chen, Aoxiang Zhou, Chunpei Li, Youquan Xian, Peng Liu, Xianxian LiAbstract
As large language models continue to develop and expand, the extensive public data they rely on faces the risk of depletion. Consequently, leveraging private data within organizations to enhance the performance of large models has emerged as a key challenge. The federated learning paradigm, combined with model fine-tuning techniques, effectively reduces the number of trainable parameters. However,the necessity to process high-dimensional feature spaces results in substantial overall computational overhead. To address this issue, we propose the Implicit Federated In-Context Learning (IFed-ICL) framework. IFed-ICL draws inspiration from federated learning to establish a novel distributed collaborative paradigm, by converting client local context examples into implicit vector representations, it enables distributed collaborative computation during the inference phase and injects model residual streams to enhance model performance. Experiments demonstrate that our proposed method achieves outstanding performance across multiple text classification tasks. Compared to traditional methods, IFed-ICL avoids the extensive parameter updates required by conventional fine-tuning methods while reducing data transmission and local computation at the client level in federated learning. This enables efficient distributed context learning using local private-domain data, significantly improving model performance on specific tasks.
Abstract (translated)
URL
https://arxiv.org/abs/2511.06757
PDF
-
Multi-Scale Feature Fusion and Graph Neural Network Integration for Text Classification with Large Language Models
2025-11-07 22:54:26Xiangchen Song, Yulin Huang, Jinxu Guo, Yuchen Liu, Yaxuan LuanAbstract
This study investigates a hybrid method for text classification that integrates deep feature extraction from large language models, multi-scale fusion through feature pyramids, and structured modeling with graph neural networks to enhance performance in complex semantic contexts. First, the large language model captures contextual dependencies and deep semantic representations of the input text, providing a rich feature foundation for subsequent modeling. Then, based on multi-level feature representations, the feature pyramid mechanism effectively integrates semantic features of different scales, balancing global information and local details to construct hierarchical semantic expressions. Furthermore, the fused features are transformed into graph representations, and graph neural networks are employed to capture latent semantic relations and logical dependencies in the text, enabling comprehensive modeling of complex interactions among semantic units. On this basis, the readout and classification modules generate the final category predictions. The proposed method demonstrates significant advantages in robustness alignment experiments, outperforming existing models on ACC, F1-Score, AUC, and Precision, which verifies the effectiveness and stability of the framework. This study not only constructs an integrated framework that balances global and local information as well as semantics and structure, but also provides a new perspective for multi-scale feature fusion and structured semantic modeling in text classification tasks.
Abstract (translated)
URL
https://arxiv.org/abs/2511.05752
PDF
-
Trustworthiness Calibration Framework for Phishing Email Detection Using Large Language Models
2025-11-06 18:14:44Daniyal Ganiuly, Assel SmaiylAbstract
Phishing emails continue to pose a persistent challenge to online communication, exploiting human trust and evading automated filters through realistic language and adaptive tactics. While large language models (LLMs) such as GPT-4 and LLaMA-3-8B achieve strong accuracy in text classification, their deployment in security systems requires assessing reliability beyond benchmark performance. To address this, this study introduces the Trustworthiness Calibration Framework (TCF), a reproducible methodology for evaluating phishing detectors across three dimensions: calibration, consistency, and robustness. These components are integrated into a bounded index, the Trustworthiness Calibration Index (TCI), and complemented by the Cross-Dataset Stability (CDS) metric that quantifies stability of trustworthiness across datasets. Experiments conducted on five corpora, such as SecureMail 2025, Phishing Validation 2024, CSDMC2010, Enron-Spam, and Nazario, using DeBERTa-v3-base, LLaMA-3-8B, and GPT-4 demonstrate that GPT-4 achieves the strongest overall trust profile, followed by LLaMA-3-8B and DeBERTa-v3-base. Statistical analysis confirms that reliability varies independently of raw accuracy, underscoring the importance of trust-aware evaluation for real-world deployment. The proposed framework establishes a transparent and reproducible foundation for assessing model dependability in LLM-based phishing detection.
Abstract (translated)
URL
https://arxiv.org/abs/2511.04728
PDF
-
Divide, Cache, Conquer: Dichotomic Prompting for Efficient Multi-Label LLM-Based Classification
2025-11-05 19:53:51Miko{\l}aj Langner, Jan Eliasz, Ewa Rudnicka, Jan Koco\'nAbstract
We introduce a method for efficient multi-label text classification with large language models (LLMs), built on reformulating classification tasks as sequences of dichotomic (yes/no) decisions. Instead of generating all labels in a single structured response, each target dimension is queried independently, which, combined with a prefix caching mechanism, yields substantial efficiency gains for short-text inference without loss of accuracy. To demonstrate the approach, we focus on affective text analysis, covering 24 dimensions including emotions and sentiment. Using LLM-to-SLM distillation, a powerful annotator model (DeepSeek-V3) provides multiple annotations per text, which are aggregated to fine-tune smaller models (HerBERT-Large, CLARIN-1B, PLLuM-8B, Gemma3-1B). The fine-tuned models show significant improvements over zero-shot baselines, particularly on the dimensions seen during training. Our findings suggest that decomposing multi-label classification into dichotomic queries, combined with distillation and cache-aware inference, offers a scalable and effective framework for LLM-based classification. While we validate the method on affective states, the approach is general and applicable across domains.
Abstract (translated)
URL
https://arxiv.org/abs/2511.03830
PDF
-
Probability-Biased Attention over Directed Bipartite Graphs for Long-Tail ICD Coding
2025-10-31 04:47:09Tianlei Chen, Yuxiao Chen, Yang Li, Feifei WangAbstract
Automated International Classification of Diseases (ICD) coding aims to assign multiple disease codes to clinical documents, constituting a crucial multi-label text classification task in healthcare informatics. However, the task is challenging due to its large label space (10,000 to 20,000 codes) and long-tail distribution, where a few codes dominate while many rare codes lack sufficient training data. To address this, we propose a learning method that models fine-grained co-occurrence relationships among codes. Specifically, we construct a Directed Bipartite Graph Encoder with disjoint sets of common and rare code nodes. To facilitate a one-way information flow, edges are directed exclusively from common to rare codes. The nature of these connections is defined by a probability-based bias, which is derived from the conditional probability of a common code co-occurring given the presence of a rare code. This bias is then injected into the encoder's attention module, a process we term Co-occurrence Encoding. This structure empowers the graph encoder to enrich rare code representations by aggregating latent comorbidity information reflected in the statistical co-occurrence of their common counterparts. To ensure high-quality input to the graph, we utilize a large language model (LLM) to generate comprehensive descriptions for codes, enriching initial embeddings with clinical context and comorbidity information, serving as external knowledge for the statistical co-occurrence relationships in the code system. Experiments on three automated ICD coding benchmark datasets demonstrate that our method achieves state-of-the-art performance with particularly notable improvements in Macro-F1, which is the key metric for long-tail classification.
Abstract (translated)
URL
https://arxiv.org/abs/2511.09559
PDF
-
SALSA: Single-pass Autoregressive LLM Structured Classification
2025-10-26 14:28:42Ruslan Berdichevsky, Shai Nahum-Gefen, Elad Ben ZakenAbstract
Despite their impressive generalization capabilities, instruction-tuned Large Language Models often underperform on text classification benchmarks. We introduce SALSA, a coherent pipeline that combines structured prompting, class-to-token mapping, and parameter-efficient fine-tuning, thereby avoiding cold-start training. Each class label is mapped to a distinct output token, and prompts are constructed to elicit a single-token response. During inference, the model's output is projected only onto the logits of the relevant class tokens, enabling efficient and accurate classification in a single forward pass. SALSA achieves state-of-the-art results across diverse benchmarks, demonstrating its robustness and scalability for LLM-based classification applications.
Abstract (translated)
URL
https://arxiv.org/abs/2510.22691
PDF
-
Weak-to-Strong Generalization under Distribution Shifts
2025-10-24 10:46:50Myeongho Jeon, Jan Sobotka, Suhwan Choi, Maria Brbi\'cAbstract
As future superhuman models become increasingly complex, accurately supervising their behavior may exceed human capabilities. Recent works have demonstrated that in such scenarios, weak models can effectively supervise strong models, a phenomenon known as weak-to-strong generalization. However, we find that naive weak-to-strong generalization fails under distribution shifts, often leading to worse performance of the strong model than its weak supervisors. To address this, we propose RAVEN, a robust weak-to-strong generalization framework that dynamically learns the optimal combinations of weak models in addition to parameters of the strong model. We demonstrate the effectiveness of RAVEN on image classification, text classification, and preference alignment tasks. RAVEN outperforms alternative baselines by over 30% on out-of-distribution tasks while matching or surpassing existing methods on in-distribution tasks. Moreover, our results show that RAVEN assigns higher weights to more accurate weak models, demonstrating its ability to automatically identify trustworthy supervision.
Abstract (translated)
URL
https://arxiv.org/abs/2510.21332
PDF
-
Framework for Machine Evaluation of Reasoning Completeness in Large Language Models For Classification Tasks
2025-10-23 20:22:22Avinash PatilAbstract
The growing adoption of machine learning (ML) in sensitive domains has heightened the demand for transparent and interpretable artificial intelligence. Large Language Models (LLMs) are increasingly capable of producing natural language explanations, yet it remains unclear whether these rationales faithfully capture the predictive signals that underlie decisions. This paper introduces RACE-Reasoning Alignment for Completeness of Explanations, a systematic framework to evaluate the alignment between LLM-generated explanations and interpretable feature importance scores derived from a logistic regression baseline. We analyze four widely used text classification datasets-WIKI ONTOLOGY, AG NEWS, IMDB, and GOEMOTIONS-and compare LLM rationales against top-ranked supporting and contradicting lexical features. To capture alignment at multiple levels of granularity, RACE implements token-aware, exact string, and edit-distance matching techniques. Empirical results reveal a consistent asymmetry: correct predictions exhibit higher coverage of supporting features, while incorrect predictions are associated with elevated coverage of contradicting features. Edit-distance matching further uncovers paraphrastic overlaps, boosting coverage while preserving this asymmetry. These findings demonstrate that LLM rationales combine both surface-level and flexible evidence reuse, yet can also amplify misleading cues in error cases. RACE provides new insights into the faithfulness of LLM explanations and establishes a quantitative basis for evaluating reasoning completeness in neural language models.
Abstract (translated)
URL
https://arxiv.org/abs/2510.21884
PDF
-
Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers
2025-10-21 07:26:38Firas Gabetni, Giuseppe Curci, Andrea Pilzer, Subhankar Roy, Elisa Ricci, Gianni FranchiAbstract
Uncertainty quantification (UQ) is essential for deploying deep neural networks in safety-critical settings. Although methods like Deep Ensembles achieve strong UQ performance, their high computational and memory costs hinder scalability to large models. We introduce Hydra Ensembles, an efficient transformer-based ensemble that prunes attention heads to create diverse members and merges them via a new multi-head attention with grouped fully-connected layers. This yields a compact model with inference speed close to a single network, matching or surpassing Deep Ensembles in UQ performance without retraining from scratch. We also provide an in-depth analysis of pruning, showing that naive approaches can harm calibration, whereas Hydra Ensembles preserves robust uncertainty. Experiments on image and text classification tasks, with various architectures, show consistent gains over Deep Ensembles. Remarkably, in zero-shot classification on ImageNet-1k, our approach surpasses state of the art methods, even without requiring additional training.
Abstract (translated)
不确定性量化（UQ）对于在安全关键环境中部署深度神经网络至关重要。尽管像Deep Ensembles这样的方法能够实现强大的不确定性量化性能，但它们的高计算和内存成本阻碍了其在大型模型中的可扩展性。我们引入了一种高效的基于Transformer的集合——Hydra Ensembles，它通过修剪注意力头来创建多样化的成员，并利用一种新的具有分组全连接层的多头注意力机制将这些成员合并起来。这种方法生成了一个紧凑的模型，其推理速度接近单个网络的速度，在不从头开始重新训练的情况下，不确定性量化的性能可以与Deep Ensembles相匹配甚至超越。 我们还提供了对修剪方法的深入分析，表明简单的修剪方式可能会损害校准效果，而Hydra Ensembles则能够保持强大的不确定性估计。在图像和文本分类任务中进行的各种架构实验显示，与Deep Ensembles相比，我们的方法具有持续的优势。尤其值得注意的是，在零样本（zero-shot）ImageNet-1k分类测试中，即使不增加额外的训练数据，我们提出的方法也超越了现有的最佳方法。
URL
https://arxiv.org/abs/2510.18358
PDF
-
Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review
2025-10-18 22:46:53Zhyar Rzgar K. Rostam, G\'abor Kert\'eszAbstract
The exponential increase in scientific literature and online information necessitates efficient methods for extracting knowledge from textual data. Natural language processing (NLP) plays a crucial role in addressing this challenge, particularly in text classification tasks. While large language models (LLMs) have achieved remarkable success in NLP, their accuracy can suffer in domain-specific contexts due to specialized vocabulary, unique grammatical structures, and imbalanced data distributions. In this systematic literature review (SLR), we investigate the utilization of pre-trained language models (PLMs) for domain-specific text classification. We systematically review 41 articles published between 2018 and January 2024, adhering to the PRISMA statement (preferred reporting items for systematic reviews and meta-analyses). This review methodology involved rigorous inclusion criteria and a multi-step selection process employing AI-powered tools. We delve into the evolution of text classification techniques and differentiate between traditional and modern approaches. We emphasize transformer-based models and explore the challenges and considerations associated with using LLMs for domain-specific text classification. Furthermore, we categorize existing research based on various PLMs and propose a taxonomy of techniques used in the field. To validate our findings, we conducted a comparative experiment involving BERT, SciBERT, and BioBERT in biomedical sentence classification. Finally, we present a comparative study on the performance of LLMs in text classification tasks across different domains. In addition, we examine recent advancements in PLMs for domain-specific text classification and offer insights into future directions and limitations in this rapidly evolving domain.
Abstract (translated)
科学文献和在线信息的指数增长要求高效的方法从文本数据中提取知识。自然语言处理（NLP）在应对这一挑战方面发挥着关键作用，特别是在文本分类任务中。尽管大型语言模型（LLMs）在NLP领域取得了显著的成功，但在特定领域的背景下，由于专业词汇、独特的语法结构以及不平衡的数据分布等原因，它们的准确性可能会受到影响。在这次系统性文献综述（SLR）中，我们研究了预训练语言模型（PLMs）在特定领域文本分类中的应用。我们在2018年至2024年1月间共审查了41篇论文，并遵循PRISMA声明（系统评价和元分析的首选报告项目）。该综述方法采用了严格的纳入标准以及使用AI工具进行多步骤选择的过程。我们深入探讨了文本分类技术的发展历程，区分了传统方法与现代方法的区别。我们强调基于转换器的模型，并研究了在特定领域内应用LLMs时面临的挑战和考虑因素。此外，我们依据不同的PLMs对现有研究进行了分类，并提出了一种该领域的技术分类体系。为了验证我们的发现，我们在生物医学句子分类任务中进行了BERT、SciBERT和BioBERT之间的比较实验。最后，我们提供了不同领域中大型语言模型在文本分类任务性能上的对比分析。此外，我们还考察了PLMs在特定领域文本分类中的最新进展，并为该快速发展的领域的未来方向及限制性因素提供见解。
URL
https://arxiv.org/abs/2510.17892
PDF
-
ProtoSiTex: Learning Semi-Interpretable Prototypes for Multi-label Text Classification
2025-10-14 13:59:28Utsav Kumar Nareti, Suraj Kumar, Soumya Pandey, Soumi Chattopadhyay, Chandranath AdakAbstract
The surge in user-generated reviews has amplified the need for interpretable models that can provide fine-grained insights. Existing prototype-based models offer intuitive explanations but typically operate at coarse granularity (sentence or document level) and fail to address the multi-label nature of real-world text classification. We propose ProtoSiTex, a semi-interpretable framework designed for fine-grained multi-label text classification. ProtoSiTex employs a dual-phase alternating training strategy: an unsupervised prototype discovery phase that learns semantically coherent and diverse prototypes, and a supervised classification phase that maps these prototypes to class labels. A hierarchical loss function enforces consistency across sub-sentence, sentence, and document levels, enhancing interpretability and alignment. Unlike prior approaches, ProtoSiTex captures overlapping and conflicting semantics using adaptive prototypes and multi-head attention. We also introduce a benchmark dataset of hotel reviews annotated at the sub-sentence level with multiple labels. Experiments on this dataset and two public benchmarks (binary and multi-class) show that ProtoSiTex achieves state-of-the-art performance while delivering faithful, human-aligned explanations, establishing it as a robust solution for semi-interpretable multi-label text classification.
Abstract (translated)
用户生成的评论数量激增，增强了对能够提供细致洞察的可解释模型的需求。现有的基于原型的模型提供了直观的解释，但通常以粗粒度（句子或文档级别）运行，并且无法解决现实世界文本分类中的多标签特性问题。我们提出了ProtoSiTex，这是一种面向细粒度多标签文本分类设计的半可解释框架。ProtoSiTex采用了一种双阶段交替训练策略：一个无监督原型发现阶段，该阶段学习语义上连贯且多样化的原型；以及一个有监督分类阶段，将这些原型映射到类别标签。层次化损失函数确保了在子句、句子和文档级别之间的一致性，从而增强了可解释性和对齐度。与先前的方法不同，ProtoSiTex使用自适应原型和多头注意力机制来捕捉重叠和冲突的语义。 我们还引入了一个基准数据集，该数据集包含酒店评论，并且这些评论在子句级别的多个标签上进行了标注。实验结果表明，在该数据集以及两个公开基准（二元分类和多类分类）上的测试中，ProtoSiTex实现了最先进的性能并提供了忠实、与人类一致的解释，从而确立了其作为半可解释多标签文本分类领域中的稳健解决方案的地位。
URL
https://arxiv.org/abs/2510.12534
PDF
-
Investigating Large Language Models' Linguistic Abilities for Text Preprocessing
2025-10-13 14:53:44Marco Braga, Gian Carlo Milanese, Gabriella PasiAbstract
Text preprocessing is a fundamental component of Natural Language Processing, involving techniques such as stopword removal, stemming, and lemmatization to prepare text as input for further processing and analysis. Despite the context-dependent nature of the above techniques, traditional methods usually ignore contextual information. In this paper, we investigate the idea of using Large Language Models (LLMs) to perform various preprocessing tasks, due to their ability to take context into account without requiring extensive language-specific annotated resources. Through a comprehensive evaluation on web-sourced data, we compare LLM-based preprocessing (specifically stopword removal, lemmatization and stemming) to traditional algorithms across multiple text classification tasks in six European languages. Our analysis indicates that LLMs are capable of replicating traditional stopword removal, lemmatization, and stemming methods with accuracies reaching 97%, 82%, and 74%, respectively. Additionally, we show that ML algorithms trained on texts preprocessed by LLMs achieve an improvement of up to 6% with respect to the $F_1$ measure compared to traditional techniques. Our code, prompts, and results are publicly available at this https URL.
Abstract (translated)
文本预处理是自然语言处理中的一个基本组成部分，涉及诸如停用词去除、词干提取和词形还原等技术，以准备作为进一步处理和分析输入的文本。尽管上述技术具有上下文依赖性，传统方法通常会忽略上下文信息。在本文中，我们探讨了使用大型语言模型（LLM）执行各种预处理任务的想法，因为它们能够在不需大量特定语言注释资源的情况下考虑到上下文。通过在六种欧洲语言的多个文本分类任务上的网络数据进行全面评估，我们将基于LLM的预处理（特别是停用词去除、词形还原和词干提取）与传统算法进行了比较。我们的分析表明，LLMs能够以高达97%、82%和74%的准确率复制传统的停用词去除、词形还原和词干提取方法。此外，我们还展示了在使用LLM预处理文本上进行训练的机器学习算法，在$F_1$度量方面相比传统技术提高了最多6%。 我们的代码、提示以及结果可在以下网址公开获取：[此链接](https://example.com)（请注意实际发表时请替换为正确的URL）。
URL
https://arxiv.org/abs/2510.11482
PDF
-
A Multi-lingual Dataset of Classified Paragraphs from Open Access Scientific Publications
2025-10-13 13:10:47Eric JeangirardAbstract
We present a dataset of 833k paragraphs extracted from CC-BY licensed scientific publications, classified into four categories: acknowledgments, data mentions, software/code mentions, and clinical trial mentions. The paragraphs are primarily in English and French, with additional European languages represented. Each paragraph is annotated with language identification (using fastText) and scientific domain (from OpenAlex). This dataset, derived from the French Open Science Monitor corpus and processed using GROBID, enables training of text classification models and development of named entity recognition systems for scientific literature mining. The dataset is publicly available on HuggingFace this https URL under a CC-BY license.
Abstract (translated)
URL
https://arxiv.org/abs/2510.21762
PDF
-
One Size Does Not Fit All: Exploring Variable Thresholds for Distance-Based Multi-Label Text Classification
2025-10-13 08:52:14Jens Van Nooten, Andriy Kosar, Guy De Pauw, Walter DaelemansAbstract
Distance-based unsupervised text classification is a method within text classification that leverages the semantic similarity between a label and a text to determine label relevance. This method provides numerous benefits, including fast inference and adaptability to expanding label sets, as opposed to zero-shot, few-shot, and fine-tuned neural networks that require re-training in such cases. In multi-label distance-based classification and information retrieval algorithms, thresholds are required to determine whether a text instance is "similar" to a label or query. Similarity between a text and label is determined in a dense embedding space, usually generated by state-of-the-art sentence encoders. Multi-label classification complicates matters, as a text instance can have multiple true labels, unlike in multi-class or binary classification, where each instance is assigned only one label. We expand upon previous literature on this underexplored topic by thoroughly examining and evaluating the ability of sentence encoders to perform distance-based classification. First, we perform an exploratory study to verify whether the semantic relationships between texts and labels vary across models, datasets, and label sets by conducting experiments on a diverse collection of realistic multi-label text classification (MLTC) datasets. We find that similarity distributions show statistically significant differences across models, datasets and even label sets. We propose a novel method for optimizing label-specific thresholds using a validation set. Our label-specific thresholding method achieves an average improvement of 46% over normalized 0.5 thresholding and outperforms uniform thresholding approaches from previous work by an average of 14%. Additionally, the method demonstrates strong performance even with limited labeled examples.
Abstract (translated)
基于距离的无监督文本分类是文本分类领域的一种方法，该方法利用标签与文本之间的语义相似性来判断标签的相关性。相比于零样本、小样本和微调神经网络需要重新训练的情况，这种方法提供了许多优势，包括快速推理以及对不断扩大的标签集具有良好的适应能力。 在多标签距离基于的分类及信息检索算法中，为了确定某个文本实例是否与一个标签或查询“相似”，都需要设定阈值。文本和标签之间的相似性是在由先进句子编码器生成的稠密嵌入空间内进行判定的。与多类或者二元分类不同，在多标签分类中，一个文本实例可能有多个真实标签。 我们在之前较少探索的主题上进行了拓展研究，详细地考察并评估了句子编码器在基于距离的分类中的性能。首先，我们通过一系列实验来验证模型、数据集和标签集合之间的语义关系是否存在差异。这些实验是在一组多样化的现实多标签文本分类（MLTC）数据集上进行的。我们发现相似性的分布统计上显著地不同，这表明了在不同的模型和数据集中存在着变化。 基于这一观察，我们提出了一个新的方法来利用验证集优化特定于标签的阈值。通过这种方法，相较于标准化后的0.5阈值法，我们的特定标签阈值设定平均提升了46%，并且相比于以往研究中的统一阈值设定策略，该方法也表现出平均14%的优势。此外，此方法即使在标注样本有限的情况下也能展现出强大的性能。 这种方法不仅提供了改进的分类准确性，还为复杂的多标签场景提供了解决方案，并且证明了针对不同数据集和标签优化特定阈值的重要性。
URL
https://arxiv.org/abs/2510.11160
PDF
