Text_Classification
-
PLEX: Perturbation-free Local Explanations for LLM-Based Text Classification
2025-07-12 06:31:38Yogachandran Rahulamathavan, Misbah Farooq, Varuna De SilvaAbstract
Large Language Models (LLMs) excel in text classification, but their complexity hinders interpretability, making it difficult to understand the reasoning behind their predictions. Explainable AI (XAI) methods like LIME and SHAP offer local explanations by identifying influential words, but they rely on computationally expensive perturbations. These methods typically generate thousands of perturbed sentences and perform inferences on each, incurring a substantial computational burden, especially with LLMs. To address this, we propose \underline{P}erturbation-free \underline{L}ocal \underline{Ex}planation (PLEX), a novel method that leverages the contextual embeddings extracted from the LLM and a ``Siamese network" style neural network trained to align with feature importance scores. This one-off training eliminates the need for subsequent perturbations, enabling efficient explanations for any new sentence. We demonstrate PLEX's effectiveness on four different classification tasks (sentiment, fake news, fake COVID-19 news and depression), showing more than 92\% agreement with LIME and SHAP. Our evaluation using a ``stress test" reveals that PLEX accurately identifies influential words, leading to a similar decline in classification accuracy as observed with LIME and SHAP when these words are removed. Notably, in some cases, PLEX demonstrates superior performance in capturing the impact of key features. PLEX dramatically accelerates explanation, reducing time and computational overhead by two and four orders of magnitude, respectively. This work offers a promising solution for explainable LLM-based text classification.
Abstract (translated)
大型语言模型（LLMs）在文本分类任务中表现出色，但它们的复杂性阻碍了可解释性，使得理解其预测背后的逻辑变得困难。像LIME和SHAP这样的可解释人工智能(XAI)方法通过识别有影响力的词汇来提供局部解释，但是这些方法依赖于计算成本高昂的数据扰动。通常情况下，为了生成成千上万个扰动的句子并对其进行推理，这种方法会产生巨大的计算负担，尤其是对于LLMs而言。 为了解决这个问题，我们提出了一种名为“无扰动局部解释”（PLEX）的新方法。PLEX利用从大型语言模型中提取的上下文嵌入和一种类似“Siamese网络”的神经网络来与特征重要性得分对齐。这种方法通过一次性的训练消除了后续需要进行扰动的需求，从而能够为任何新的句子提供高效的解释。 我们在四个不同的分类任务（情感分析、假新闻识别、新冠假新闻检测以及抑郁症检测）上展示了PLEX的有效性，并且PLEX在92%以上的测试中与LIME和SHAP达成一致。通过使用“压力测试”进行评估，PLEX准确地识别出有影响力的词汇，在这些单词被移除后导致分类准确性下降的情况类似于LIME和SHAP的表现。 值得注意的是，在某些情况下，PLEX展示了捕捉关键特征影响的优越性能。更重要的是，PLEX极大地加快了解释过程，将时间和计算开销分别减少了两个和四个数量级。这项工作为基于LLM的文本分类提供了可解释性的有希望的解决方案。
URL
https://arxiv.org/abs/2507.10596
PDF
-
GNN-CNN: An Efficient Hybrid Model of Convolutional and Graph Neural Networks for Text Representation
2025-07-10 04:13:53Fardin RastakhizAbstract
Time, cost, and energy efficiency are critical considerations in Deep-Learning (DL), particularly when processing long texts. Transformers, which represent the current state of the art, exhibit quadratic computational complexity relative to input length, making them inefficient for extended documents. This study introduces a novel model architecture that combines Graph Neural Networks (GNNs) and Convolutional Neural Networks (CNNs), integrated with a real-time, end-to-end graph generation mechanism. The model processes compact batches of character-level inputs without requiring padding or truncation. To enhance performance while maintaining high speed and efficiency, the model incorporates information from Large Language Models (LLMs), such as token embeddings and sentiment polarities, through efficient dictionary lookups. It captures local contextual patterns using CNNs, expands local receptive fields via lattice-based graph structures, and employs small-world graphs to aggregate document-level information. The generated graphs exhibit structural properties indicative of meaningful semantic organization, with an average clustering coefficient of approximately 0.45 and an average shortest path length ranging between 4 and 5. The model is evaluated across multiple text classification tasks, including sentiment analysis and news-categorization, and is compared against state-of-the-art models. Experimental results confirm the proposed model's efficiency and competitive performance.
Abstract (translated)
时间、成本和能源效率是深度学习（DL）中的关键考虑因素，尤其是在处理长文本时。当前最先进的模型——变压器表现出与输入长度呈二次关系的计算复杂性，这使得它们在处理较长文档时效率低下。本研究提出了一种结合图神经网络（GNNs）和卷积神经网络（CNNs）的新颖架构，并整合了实时、端到端的图生成机制。该模型以字符级输入的小批量为单位进行处理，无需填充或截断。 为了在保持高速度和高效率的同时提升性能，该模型通过高效的字典查找方式集成了大型语言模型（LLMs）的信息，包括token嵌入和情感极性等。具体来说，它使用CNN捕捉局部上下文模式，并利用基于格子的图结构扩展了局部感受野；同时采用小世界图来汇总文档级别的信息。 生成的图表显示有意义的语义组织特征，其平均聚类系数约为0.45，而最短路径长度则在4到5之间。该模型在多项文本分类任务上进行了评估，包括情感分析和新闻分类，并与其他最先进的模型进行了比较。实验结果证实了所提出模型的有效性和竞争性性能。
URL
https://arxiv.org/abs/2507.07414
PDF
-
Unifying Re-Identification, Attribute Inference, and Data Reconstruction Risks in Differential Privacy
2025-07-09 15:59:30Bogdan Kulynych, Juan Felipe Gomez, Georgios Kaissis, Jamie Hayes, Borja Balle, Flavio du Pin Calmon, Jean Louis RaisaroAbstract
Differentially private (DP) mechanisms are difficult to interpret and calibrate because existing methods for mapping standard privacy parameters to concrete privacy risks -- re-identification, attribute inference, and data reconstruction -- are both overly pessimistic and inconsistent. In this work, we use the hypothesis-testing interpretation of DP ($f$-DP), and determine that bounds on attack success can take the same unified form across re-identification, attribute inference, and data reconstruction risks. Our unified bounds are (1) consistent across a multitude of attack settings, and (2) tunable, enabling practitioners to evaluate risk with respect to arbitrary (including worst-case) levels of baseline risk. Empirically, our results are tighter than prior methods using $\varepsilon$-DP, Rényi DP, and concentrated DP. As a result, calibrating noise using our bounds can reduce the required noise by 20% at the same risk level, which yields, e.g., more than 15pp accuracy increase in a text classification task. Overall, this unifying perspective provides a principled framework for interpreting and calibrating the degree of protection in DP against specific levels of re-identification, attribute inference, or data reconstruction risk.
Abstract (translated)
差分隐私（DP）机制难以解释和调整，因为现有的将标准隐私参数映射到具体隐私风险（如重新识别、属性推断和数据重建）的方法既过于悲观又不一致。在这项工作中，我们使用了差分隐私的假设检验解释（$f$-DP），并确定了攻击成功的界限在重新识别、属性推断和数据重建风险之间可以采用统一的形式。我们的统一界线具有以下特点：(1) 在多种攻击设置下保持一致性；(2) 可调，使从业者能够根据任意基准风险水平评估风险（包括最坏情况）。从实证角度看，与使用$\varepsilon$-DP、Rényi DP和集中化DP的方法相比，我们的结果更为精确。因此，在相同的风险级别下，利用这些界限校准噪声可以将所需的噪声减少20%，从而例如在文本分类任务中提高超过15个百分点的准确率。总体而言，这种统一的观点提供了一个原则性的框架来解释和调整差分隐私机制以抵御特定级别的重新识别、属性推断或数据重建风险的程度。
URL
https://arxiv.org/abs/2507.06969
PDF
-
Verified Language Processing with Hybrid Explainability: A Technical Report
2025-07-07 14:00:05Oliver Robert Fox, Giacomo Bergami, Graham MorganAbstract
The volume and diversity of digital information have led to a growing reliance on Machine Learning techniques, such as Natural Language Processing, for interpreting and accessing appropriate data. While vector and graph embeddings represent data for similarity tasks, current state-of-the-art pipelines lack guaranteed explainability, failing to determine similarity for given full texts accurately. These considerations can also be applied to classifiers exploiting generative language models with logical prompts, which fail to correctly distinguish between logical implication, indifference, and inconsistency, despite being explicitly trained to recognise the first two classes. We present a novel pipeline designed for hybrid explainability to address this. Our methodology combines graphs and logic to produce First-Order Logic representations, creating machine- and human-readable representations through Montague Grammar. Preliminary results indicate the effectiveness of this approach in accurately capturing full text similarity. To the best of our knowledge, this is the first approach to differentiate between implication, inconsistency, and indifference for text classification tasks. To address the limitations of existing approaches, we use three self-contained datasets annotated for the former classification task to determine the suitability of these approaches in capturing sentence structure equivalence, logical connectives, and spatiotemporal reasoning. We also use these data to compare the proposed method with language models pre-trained for detecting sentence entailment. The results show that the proposed method outperforms state-of-the-art models, indicating that natural language understanding cannot be easily generalised by training over extensive document corpora. This work offers a step toward more transparent and reliable Information Retrieval from extensive textual data.
Abstract (translated)
数字信息的量和多样性导致了对机器学习技术（如自然语言处理）在解释和访问适当数据方面的日益依赖。虽然向量和图嵌入可以表示用于相似性任务的数据，但当前最先进的管道缺乏可保证的透明度，在确定给定全文的相似性方面不够准确。这些考虑同样适用于利用逻辑提示的生成式语言模型分类器，尽管它们被明确训练来识别前两类，但在区分逻辑蕴含、无关性和不一致性时仍会失败。 我们提出了一种新型管道，旨在通过混合可解释性解决上述问题。我们的方法结合了图和逻辑来产生一阶逻辑表示，并通过Montague语法生成机器和人类都易于理解的表示形式。初步结果显示，这种方法在准确捕捉全文相似性方面是有效的。据我们所知，这是第一个能够在文本分类任务中区分蕴含、不一致性和无关性的方法。 为了解决现有方法的局限性，我们使用了三个自包含的数据集，这些数据集被注释用于上述分类任务，以确定这些方法在捕捉句子结构等价性、逻辑连接词和时空推理方面的能力。同时，我们也利用这些数据来将提出的模型与预训练的语言模型进行对比，后者旨在检测句子蕴含。 实验结果表明，我们提出的方法优于最先进的模型，这表明自然语言理解不能通过大规模文档语料库的培训轻易实现泛化。这项工作朝着从大量文本数据中获取更透明和可靠的信息检索迈出了重要一步。
URL
https://arxiv.org/abs/2507.05017
PDF
-
Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification
2025-07-07 13:48:54Chenfei Xiong, Jingwei Ni, Yu Fan, Vil\'em Zouhar, Donya Rooein, Lorena Calvo-Bartolom\'e, Alexander Hoyle, Zhijing Jin, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, Mennatallah El-Assady, Elliott AshAbstract
We introduce Co-DETECT (Collaborative Discovery of Edge cases in TExt ClassificaTion), a novel mixed-initiative annotation framework that integrates human expertise with automatic annotation guided by large language models (LLMs). Co-DETECT starts with an initial, sketch-level codebook and dataset provided by a domain expert, then leverages the LLM to annotate the data and identify edge cases that are not well described by the initial codebook. Specifically, Co-DETECT flags challenging examples, induces high-level, generalizable descriptions of edge cases, and assists user in incorporating edge case handling rules to improve the codebook. This iterative process enables more effective handling of nuanced phenomena through compact, generalizable annotation rules. Extensive user study, qualitative and quantitative analyses prove the effectiveness of Co-DETECT.
Abstract (translated)
我们介绍了Co-DETECT（文本分类中边缘情况的协作发现），这是一个新颖的混合主动注释框架，它结合了人类专业知识与大型语言模型（LLMs）指导下的自动标注。Co-DETECT始于领域专家提供的初步代码本和数据集，然后利用LLM对数据进行标注，并识别出那些初始代码本描述不充分的边缘案例。具体来说，Co-DETECT会标记具有挑战性的例子，引出高层次、可泛化的边缘案例描述，并帮助用户将处理边缘案例的规则纳入代码本中以改进其内容。这一迭代过程使得通过简洁且通用的注释规则更有效地处理细微现象成为可能。广泛的用户研究以及定性和定量分析证明了Co-DETECT的有效性。
URL
https://arxiv.org/abs/2507.05010
PDF
-
Bridging KAN and MLP: MJKAN, a Hybrid Architecture with Both Efficiency and Expressiveness
2025-07-07 06:13:32Hanseon Joo, Hayoung Choi, Ook Lee, Minjong CheonAbstract
Kolmogorov-Arnold Networks (KANs) have garnered attention for replacing fixed activation functions with learnable univariate functions, but they exhibit practical limitations, including high computational costs and performance deficits in general classification tasks. In this paper, we propose the Modulation Joint KAN (MJKAN), a novel neural network layer designed to overcome these challenges. MJKAN integrates a FiLM (Feature-wise Linear Modulation)-like mechanism with Radial Basis Function (RBF) activations, creating a hybrid architecture that combines the non-linear expressive power of KANs with the efficiency of Multilayer Perceptrons (MLPs). We empirically validated MJKAN's performance across a diverse set of benchmarks, including function regression, image classification (MNIST, CIFAR-10/100), and natural language processing (AG News, SMS Spam). The results demonstrate that MJKAN achieves superior approximation capabilities in function regression tasks, significantly outperforming MLPs, with performance improving as the number of basis functions increases. Conversely, in image and text classification, its performance was competitive with MLPs but revealed a critical dependency on the number of basis functions. We found that a smaller basis size was crucial for better generalization, highlighting that the model's capacity must be carefully tuned to the complexity of the data to prevent overfitting. In conclusion, MJKAN offers a flexible architecture that inherits the theoretical advantages of KANs while improving computational efficiency and practical viability.
Abstract (translated)
Kolmogorov-Arnold Networks（KAN）因其用可学习的一元函数替代固定激活函数而受到关注，但它们在实际应用中存在一些限制，包括计算成本高和一般分类任务中的性能不足。本文提出了一种新的神经网络层——调制联合KAN（MJKAN），旨在克服这些挑战。MJKAN结合了类似于FiLM（按特征线性调制）的机制和径向基函数（RBF）激活，构建了一个混合架构，它融合了KAN的非线性表达能力和多层感知机（MLP）的效率。 我们通过一系列基准测试对MJKAN的表现进行了实证验证，包括函数回归、图像分类（MNIST, CIFAR-10/100）和自然语言处理（AG News, SMS Spam）。结果表明，在函数回归任务中，MJKAN展示了优越的逼近能力，并显著超越了MLP，且随着基函数数量的增加性能有所提升。然而，在图像和文本分类中，其表现与MLP相当，但显现出对基函数数量的关键依赖性。我们发现较小的基础大小对于更好的泛化至关重要，这表明模型容量需要根据数据复杂度进行仔细调整以防止过拟合。 综上所述，MJKAN提供了一种灵活的架构，继承了KAN理论上的优势，同时在计算效率和实际可行性方面进行了改进。
URL
https://arxiv.org/abs/2507.04690
PDF
-
Assessing the Capabilities and Limitations of FinGPT Model in Financial NLP Applications
2025-07-06 20:02:08Prudence Djagba, Chimezie A. OdinakachukwuAbstract
This work evaluates FinGPT, a financial domain-specific language model, across six key natural language processing (NLP) tasks: Sentiment Analysis, Text Classification, Named Entity Recognition, Financial Question Answering, Text Summarization, and Stock Movement Prediction. The evaluation uses finance-specific datasets to assess FinGPT's capabilities and limitations in real-world financial applications. The results show that FinGPT performs strongly in classification tasks such as sentiment analysis and headline categorization, often achieving results comparable to GPT-4. However, its performance is significantly lower in tasks that involve reasoning and generation, such as financial question answering and summarization. Comparisons with GPT-4 and human benchmarks highlight notable performance gaps, particularly in numerical accuracy and complex reasoning. Overall, the findings indicate that while FinGPT is effective for certain structured financial tasks, it is not yet a comprehensive solution. This research provides a useful benchmark for future research and underscores the need for architectural improvements and domain-specific optimization in financial language models.
Abstract (translated)
这项工作评估了FinGPT，这是一种专门针对金融领域的语言模型，并对其在六个关键自然语言处理（NLP）任务中的表现进行了评价：情感分析、文本分类、命名实体识别、财务问答、文本摘要和股票价格走势预测。该评估使用了特定于金融的数据集来考察FinGPT在现实世界金融应用中的能力和局限性。 结果显示，FinGPT在诸如情感分析和新闻标题分类等分类任务中表现出色，并经常达到与GPT-4相媲美的结果水平。然而，在涉及推理和生成的任务（如财务问答和摘要生成）方面，它的表现显著较低。与GPT-4及人类基准的比较表明了明显的性能差距，特别是在数值准确性和复杂推理能力上。 总体而言，研究发现表明尽管FinGPT在某些结构化的金融任务中是有效的，但它还不是一个全面的解决方案。这项研究为未来的研究提供了有用的基准，并强调了对金融语言模型架构改进和领域特定优化的需求。
URL
https://arxiv.org/abs/2507.08015
PDF
-
Two-Stage Reasoning-Infused Learning: Improving Classification with LLM-Generated Reasoning
2025-06-30 19:34:57Mads Henrichsen, Rasmus KrebsAbstract
Standard classification models often map inputs directly to labels without explicit reasoning, potentially limiting their performance, robustness, and interpretability. This paper introduces a novel two-stage approach to enhance text classification by leveraging Large Language Model (LLM)-generated reasonings. In the first stage, we fine-tune a Llama-3.2-1B-Instruct model (henceforth Llama-R-Gen) on a general-purpose reasoning dataset (syvai/reasoning-gen) to generate textual reasoning (R) given a question and its answer. In the second stage, this generally trained Llama-R-Gen is used offline to create an augmented training dataset for a downstream generative model. This downstream model, based on Llama-3.2-1B-Instruct, takes only the input text (Q) and is trained to output the generated reasoning (R) immediately followed by the predicted emotion (A). We demonstrate this methodology on the dair-ai/emotion dataset for emotion classification. Our experiments show that the generative model trained to output reasoning and the emotion (Classifier Q->RA) achieves a significant improvement of 8.7 percentage points in accuracy (for emotion prediction) compared to a baseline generative model trained solely to output the emotion (Classifier Q->A), highlighting the strong generalization capabilities of the reasoning generation and the benefit of explicit reasoning training. This work underscores the potential of LLM-generated reasonings for creating richer training datasets, thereby improving the performance of diverse downstream NLP tasks and providing explicit explanations.
Abstract (translated)
标准的分类模型通常直接将输入映射到标签，而不进行显式的推理过程，这可能会限制其性能、鲁棒性和可解释性。本文提出了一种新颖的两阶段方法，通过利用大型语言模型（LLM）生成的推理来增强文本分类。在第一阶段，我们对Llama-3.2-1B-Instruct模型（以下简称 Llama-R-Gen）进行微调，使其在一个通用推理数据集（syvai/reasoning-gen）上生成给定问题及其答案的文字推理（R）。第二阶段中，将这个经过一般训练的Llama-R-Gen离线应用于创建下游生成模型的增强型训练数据集。该下游模型基于Llama-3.2-1B-Instruct，并且仅接收输入文本（Q），其被训练以直接输出由LLM生成的推理（R）紧接预测的情感标签（A）。我们在dair-ai/emotion情感分类数据集上展示了这种方法的应用。实验结果表明，与仅用于输出情感标签（Classifier Q->A）的基础生成模型相比，训练为同时输出推理和情感（Classifier Q->RA）的生成模型，在准确率方面有显著提升（8.7个百分点），这突显了通过明确的推理训练获得的强大泛化能力以及这种训练方法的优势。这项工作强调了LLM生成推理在创建更丰富的训练数据集方面的潜力，从而提高了各种下游自然语言处理任务的表现，并提供了明确的解释支持。
URL
https://arxiv.org/abs/2507.00214
PDF
-
Unveiling Decision-Making in LLMs for Text Classification : Extraction of influential and interpretable concepts with Sparse Autoencoders
2025-06-30 15:18:50Mathis Le Bail, J\'er\'emie Dentan, Davide Buscaldi, Sonia VanierAbstract
Sparse Autoencoders (SAEs) have been successfully used to probe Large Language Models (LLMs) and extract interpretable concepts from their internal representations. These concepts are linear combinations of neuron activations that correspond to human-interpretable features. In this paper, we investigate the effectiveness of SAE-based explainability approaches for sentence classification, a domain where such methods have not been extensively explored. We present a novel SAE-based architecture tailored for text classification, leveraging a specialized classifier head and incorporating an activation rate sparsity loss. We benchmark this architecture against established methods such as ConceptShap, Independent Component Analysis, and other SAE-based concept extraction techniques. Our evaluation covers two classification benchmarks and four fine-tuned LLMs from the Pythia family. We further enrich our analysis with two novel metrics for measuring the precision of concept-based explanations, using an external sentence encoder. Our empirical results show that our architecture improves both the causality and interpretability of the extracted features.
Abstract (translated)
稀疏自编码器（SAEs）已被成功用于探测大型语言模型（LLMs），并从其内部表示中提取可解释的概念。这些概念是与人类可理解特征相对应的神经元激活线性组合。在本文中，我们研究了基于SAE的可解释方法在句子分类领域的有效性，在这一领域此类方法尚未得到广泛探索。我们提出了一种新颖的专为文本分类设计的基于SAE的架构，并利用了一个专门的分类头和激活率稀疏损失。我们将此架构与已有的方法（如ConceptShap、独立成分分析以及其他基于SAE的概念提取技术）进行了基准测试。我们的评估涵盖了两个分类基准以及来自Pythia家族的四个微调后的大型语言模型。我们进一步通过两种新的度量标准丰富了我们的分析，这些度量使用外部句子编码器来衡量基于概念的解释的精确性。实证结果表明，我们的架构提高了提取特征的原因性和可解释性。
URL
https://arxiv.org/abs/2506.23951
PDF
-
Robustness of Misinformation Classification Systems to Adversarial Examples Through BeamAttack
2025-06-30 09:37:19Arnisa Fazla, Lucas Krauter, David Guzman Piedrahita, Andrianos MichailAbstract
We extend BeamAttack, an adversarial attack algorithm designed to evaluate the robustness of text classification systems through word-level modifications guided by beam search. Our extensions include support for word deletions and the option to skip substitutions, enabling the discovery of minimal modifications that alter model predictions. We also integrate LIME to better prioritize word replacements. Evaluated across multiple datasets and victim models (BiLSTM, BERT, and adversarially trained RoBERTa) within the BODEGA framework, our approach achieves over a 99\% attack success rate while preserving the semantic and lexical similarity of the original texts. Through both quantitative and qualitative analysis, we highlight BeamAttack's effectiveness and its limitations. Our implementation is available at this https URL
Abstract (translated)
我们扩展了BeamAttack，这是一种通过词级修改（由束搜索引导）来评估文本分类系统鲁棒性的对抗性攻击算法。我们的扩展包括支持词删除和跳过替换的选项，从而能够发现能改变模型预测结果的最小化修改。此外，我们将LIME集成进来，以便更好地优先考虑单词替换。在BODEGA框架下，针对多个数据集和受害模型（BiLSTM、BERT以及对抗性训练过的RoBERTa），我们的方法实现了超过99%的成功攻击率，并同时保持了原始文本的语义和词汇相似性。通过定量和定性的分析，我们强调了BeamAttack的有效性和其局限性。我们的实现代码可在[此处](https://this https URL)获得。
URL
https://arxiv.org/abs/2506.23661
PDF
-
LinguaSynth: Heterogeneous Linguistic Signals for News Classification
2025-06-27 01:45:20Duo Zhang, Junyi MoAbstract
Deep learning has significantly advanced NLP, but its reliance on large black-box models introduces critical interpretability and computational efficiency concerns. This paper proposes LinguaSynth, a novel text classification framework that strategically integrates five complementary linguistic feature types: lexical, syntactic, entity-level, word-level semantics, and document-level semantics within a transparent logistic regression model. Unlike transformer-based architectures, LinguaSynth maintains interpretability and computational efficiency, achieving an accuracy of 84.89 percent on the 20 Newsgroups dataset and surpassing a robust TF-IDF baseline by 3.32 percent. Through rigorous feature interaction analysis, we show that syntactic and entity-level signals provide essential disambiguation and effectively complement distributional semantics. LinguaSynth sets a new benchmark for interpretable, resource-efficient NLP models and challenges the prevailing assumption that deep neural networks are necessary for high-performing text classification.
Abstract (translated)
深度学习在自然语言处理（NLP）领域取得了显著进展，但其对大型黑箱模型的依赖引发了关键的可解释性和计算效率问题。本文提出了LinguaSynth，这是一种新颖的文本分类框架，它策略性地整合了五种互补的语言特征类型：词汇、句法、实体级、词级别语义和文档级别语义，在一个透明的逻辑回归模型中使用这些特征。与基于变换器（transformer）的架构不同，LinguaSynth保持了可解释性和计算效率，在20 Newsgroups数据集上实现了84.89% 的准确率，并比坚固的TF-IDF基线高出3.32个百分点。通过严格的特征交互分析，我们证明句法和实体级信号提供了重要的消歧作用，并且有效地补充了分布语义学。LinguaSynth为可解释、资源高效的NLP模型设立了新的基准，并挑战了深度神经网络对于高性能文本分类是必要的这一普遍假设。
URL
https://arxiv.org/abs/2506.21848
PDF
-
Perspectives in Play: A Multi-Perspective Approach for More Inclusive NLP Systems
2025-06-25 07:53:36Benedetta Muscato, Lucia Passaro, Gizem Gezici, Fosca GiannottiAbstract
In the realm of Natural Language Processing (NLP), common approaches for handling human disagreement consist of aggregating annotators' viewpoints to establish a single ground truth. However, prior studies show that disregarding individual opinions can lead can lead to the side effect of underrepresenting minority perspectives, especially in subjective tasks, where annotators may systematically disagree because of their preferences. Recognizing that labels reflect the diverse backgrounds, life experiences, and values of individuals, this study proposes a new multi-perspective approach using soft labels to encourage the development of the next generation of perspective aware models, more inclusive and pluralistic. We conduct an extensive analysis across diverse subjective text classification tasks, including hate speech, irony, abusive language, and stance detection, to highlight the importance of capturing human disagreements, often overlooked by traditional aggregation methods. Results show that the multi-perspective approach not only better approximates human label distributions, as measured by Jensen-Shannon Divergence (JSD), but also achieves superior classification performance (higher F1 scores), outperforming traditional approaches. However, our approach exhibits lower confidence in tasks like irony and stance detection, likely due to the inherent subjectivity present in the texts. Lastly, leveraging Explainable AI (XAI), we explore model uncertainty and uncover meaningful insights into model predictions.
Abstract (translated)
在自然语言处理（NLP）领域，处理人类意见分歧的常见方法是汇总标注者的观点以确立单一的事实基准。然而，先前的研究表明，忽视个体意见会导致少数派视角代表性不足的问题，尤其是在主观任务中，由于个人偏好，标注者可能系统性地持有不同意见。考虑到标签反映了个人背景、生活经验和价值观的多样性，本研究提出了一种新的多视角方法，使用软标签来促进下一代视角感知模型的发展，使其更加包容和多元化。我们在包括仇恨言论、讽刺、滥用语言以及立场检测在内的多种主观文本分类任务上进行了广泛的分析，以强调捕捉人类意见分歧的重要性，这些分歧往往被传统的汇总方法所忽略。研究结果表明，多视角方法不仅在用杰森-香农散度（JSD）衡量的人类标签分布方面更接近真实情况，而且还在分类性能（更高的F1得分）上表现出色，优于传统方法。然而，在讽刺和立场检测等任务中，我们的方法显示出较低的置信度，这可能是由于文本中存在的固有主观性所致。最后，通过可解释人工智能（XAI），我们探讨了模型不确定性，并揭示了有关模型预测的重要见解。
URL
https://arxiv.org/abs/2506.20209
PDF
-
Performance of diverse evaluation metrics in NLP-based assessment and text generation of consumer complaints
2025-06-23 17:26:38Peiheng Gao, Chen Yang, Ning Sun, Ri\v{c}ardas ZitikisAbstract
Machine learning (ML) has significantly advanced text classification by enabling automated understanding and categorization of complex, unstructured textual data. However, accurately capturing nuanced linguistic patterns and contextual variations inherent in natural language, particularly within consumer complaints, remains a challenge. This study addresses these issues by incorporating human-experience-trained algorithms that effectively recognize subtle semantic differences crucial for assessing consumer relief eligibility. Furthermore, we propose integrating synthetic data generation methods that utilize expert evaluations of generative adversarial networks and are refined through expert annotations. By combining expert-trained classifiers with high-quality synthetic data, our research seeks to significantly enhance machine learning classifier performance, reduce dataset acquisition costs, and improve overall evaluation metrics and robustness in text classification tasks.
Abstract (translated)
机器学习（ML）通过实现对复杂、非结构化文本数据的自动理解和分类，在文本分类方面取得了显著进展。然而，准确捕捉自然语言中细微的语言模式和上下文变化，特别是在消费者投诉中的问题，仍然是一项挑战。本研究通过引入基于人类经验训练的算法来解决这些问题，这些算法能够有效识别评估消费者救济资格时至关重要的细微语义差异。 此外，我们还提议整合生成合成数据的方法，利用专家对生成对抗网络（GAN）的评价，并通过专家注释进行改进。通过结合由专家培训的分类器和高质量的合成数据，我们的研究旨在显著提高机器学习分类器的表现，减少数据集获取成本，并改善文本分类任务的整体评估指标和鲁棒性。
URL
https://arxiv.org/abs/2506.21623
PDF
-
Can Generated Images Serve as a Viable Modality for Text-Centric Multimodal Learning?
2025-06-21 07:32:09Yuesheng Huang, Peng Zhang, Riliang Liu, Jiaqi LiangAbstract
A significant ``modality gap" exists between the abundance of text-only data and the increasing power of multimodal models. This work systematically investigates whether images generated on-the-fly by Text-to-Image (T2I) models can serve as a valuable complementary modality for text-centric tasks. Through a comprehensive evaluation framework on text classification, we analyze the impact of critical variables, including T2I model quality, prompt engineering strategies, and multimodal fusion architectures. Our findings demonstrate that this``synthetic perception" can yield significant performance gains, even when augmenting strong large language model baselines. However, we find the effectiveness of this approach is highly conditional, depending critically on the semantic alignment between text and the generated image, the inherent ``visual groundability" of the task, and the generative fidelity of the T2I model. Our work establishes the first rigorous benchmark for this paradigm, providing a clear analysis of its potential and current limitations, and demonstrating its viability as a pathway to enrich language understanding in traditionally unimodal scenarios.
Abstract (translated)
在文本数据丰富而多模态模型能力日益增强的情况下，存在一个显著的“模式差距”。本项研究系统地探讨了通过文字生成图像（T2I）的模型即时生成的图片能否作为以文本为中心的任务的一种有价值的补充模式。我们通过全面的评估框架，在文本分类任务上分析了关键变量的影响，包括T2I模型的质量、提示工程策略和多模态融合架构。我们的研究结果表明，“合成感知”即使在增强强大的大规模语言模型基准的情况下也能带来显著的性能提升。然而，这种方法的有效性高度依赖于文本与生成图像之间的语义一致性、任务本身的“视觉基础性”，以及T2I模型的生成保真度。 本项工作建立了这一范例的第一个严格基准，对它的潜力和当前限制进行了清晰分析，并展示了其作为丰富传统单一模式场景中语言理解途径的可能性。
URL
https://arxiv.org/abs/2506.17623
PDF
-
A Comparative Study of Task Adaptation Techniques of Large Language Models for Identifying Sustainable Development Goals
2025-06-18 07:42:32Andrea Cadeddu, Alessandro Chessa, Vincenzo De Leo, Gianni Fenu, Enrico Motta, Francesco Osborne, Diego Reforgiato Recupero, Angelo Salatino, Luca SecchiAbstract
In 2012, the United Nations introduced 17 Sustainable Development Goals (SDGs) aimed at creating a more sustainable and improved future by 2030. However, tracking progress toward these goals is difficult because of the extensive scale and complexity of the data involved. Text classification models have become vital tools in this area, automating the analysis of vast amounts of text from a variety of sources. Additionally, large language models (LLMs) have recently proven indispensable for many natural language processing tasks, including text classification, thanks to their ability to recognize complex linguistic patterns and semantics. This study analyzes various proprietary and open-source LLMs for a single-label, multi-class text classification task focused on the SDGs. Then, it also evaluates the effectiveness of task adaptation techniques (i.e., in-context learning approaches), namely Zero-Shot and Few-Shot Learning, as well as Fine-Tuning within this domain. The results reveal that smaller models, when optimized through prompt engineering, can perform on par with larger models like OpenAI's GPT (Generative Pre-trained Transformer).
Abstract (translated)
2012年，联合国推出了17项可持续发展目标（SDGs），旨在通过到2030年的努力创造一个更加可持续和改善的未来。然而，由于涉及的数据规模庞大且复杂，追踪这些目标的进展情况变得十分困难。文本分类模型在这一领域中成为了至关重要的工具，能够自动分析来自各种来源的大批量文本数据。此外，大型语言模型（LLMs）最近被证明对于许多自然语言处理任务来说是不可或缺的，包括文本分类，这是因为它们具备识别复杂语言模式和语义的能力。本研究针对SDGs的一项单一标签、多类别文本分类任务，评估了多种专有及开源的大规模语言模型的性能。此外，该研究还评估了在这一领域中任务适应技术（即上下文学习方法）的有效性，具体包括零样本学习和少量样本学习，以及微调的方法。结果表明，在通过提示工程进行优化后，较小的语言模型可以与像OpenAI的GPT（生成式预训练变换器）这样的大型语言模型一样高效地工作。
URL
https://arxiv.org/abs/2506.15208
PDF
-
Identifying social isolation themes in NVDRS text narratives using topic modeling and text-classification methods
2025-06-18 00:13:42Drew Walker, Swati Rajwal, Sudeshna Das, Snigdha Peddireddy, Abeed SarkerAbstract
Social isolation and loneliness, which have been increasing in recent years strongly contribute toward suicide rates. Although social isolation and loneliness are not currently recorded within the US National Violent Death Reporting System's (NVDRS) structured variables, natural language processing (NLP) techniques can be used to identify these constructs in law enforcement and coroner medical examiner narratives. Using topic modeling to generate lexicon development and supervised learning classifiers, we developed high-quality classifiers (average F1: .86, accuracy: .82). Evaluating over 300,000 suicides from 2002 to 2020, we identified 1,198 mentioning chronic social isolation. Decedents had higher odds of chronic social isolation classification if they were men (OR = 1.44; CI: 1.24, 1.69, p<.0001), gay (OR = 3.68; 1.97, 6.33, p<.0001), or were divorced (OR = 3.34; 2.68, 4.19, p<.0001). We found significant predictors for other social isolation topics of recent or impending divorce, child custody loss, eviction or recent move, and break-up. Our methods can improve surveillance and prevention of social isolation and loneliness in the United States.
Abstract (translated)
近年来，社会孤立和孤独感的增加显著推动了自杀率的增长。尽管美国国家暴力死亡报告系统（NVDRS）当前并未在其结构化变量中记录这些因素，但自然语言处理（NLP）技术可以用来在执法部门和验尸官的描述文档中识别这些特征。通过主题建模生成词典开发和监督学习分类器，我们成功创建了高质量的分类器（平均F1分数为0.86，准确率为0.82）。评估从2002年到2020年的超过30万起自杀案例后，我们发现有1,198起案例提及慢性社会孤立。死者若为男性，则被归类于慢性社会孤立的风险更高（OR = 1.44；CI：1.24, 1.69，p<.0001），同性恋者风险也较高（OR = 3.68；1.97, 6.33，p<.0001），离婚者同样具有更高的风险（OR = 3.34；2.68, 4.19，p<.0001）。我们还发现近期或即将面临的离婚、失去子女监护权、被驱逐或近期内搬迁以及分手等事件是其他社会孤立主题的重要预测因素。我们的方法可以改进美国对社会孤立和孤独感的监测与预防措施。
URL
https://arxiv.org/abs/2506.15030
PDF
-
Medical Argument Mining: Exploitation of Scarce Data Using NLI Systems
2025-06-15 12:10:28Maitane Urruela, Sergio Mart\'in, Iker De la Iglesia, Ander BarrenaAbstract
This work presents an Argument Mining process that extracts argumentative entities from clinical texts and identifies their relationships using token classification and Natural Language Inference techniques. Compared to straightforward methods like text classification, this methodology demonstrates superior performance in data-scarce settings. By assessing the effectiveness of these methods in identifying argumentative structures that support or refute possible diagnoses, this research lays the groundwork for future tools that can provide evidence-based justifications for machine-generated clinical conclusions.
Abstract (translated)
这项工作提出了一种论点挖掘过程，该过程从临床文本中提取论据实体，并利用词元分类和自然语言推理技术来识别它们之间的关系。与诸如文本分类之类的直接方法相比，这种研究方法在数据不足的情况下表现出色。通过评估这些方法在识别支持或反驳可能诊断的论证结构方面的有效性，这项研究为未来能够提供基于证据的理由以支持机器生成的临床结论的工具奠定了基础。
URL
https://arxiv.org/abs/2506.12823
PDF
-
Generative or Discriminative? Revisiting Text Classification in the Era of Transformers
2025-06-13 19:03:58Siva Rajesh Kasa, Karan Gupta, Sumegh Roychowdhury, Ashutosh Kumar, Yaswanth Biruduraju, Santhosh Kumar Kasa, Nikhil Priyatam Pattisapu, Arindam Bhattacharya, Shailendra Agarwal, Vijay huddarAbstract
The comparison between discriminative and generative classifiers has intrigued researchers since Efron's seminal analysis of logistic regression versus discriminant analysis. While early theoretical work established that generative classifiers exhibit lower sample complexity but higher asymptotic error in simple linear settings, these trade-offs remain unexplored in the transformer era. We present the first comprehensive evaluation of modern generative and discriminative architectures - Auto-regressive modeling, Masked Language Modeling, Discrete Diffusion, and Encoders for text classification. Our study reveals that the classical 'two regimes' phenomenon manifests distinctly across different architectures and training paradigms. Beyond accuracy, we analyze sample efficiency, calibration, noise robustness, and ordinality across diverse scenarios. Our findings offer practical guidance for selecting the most suitable modeling approach based on real-world constraints such as latency and data limitations.
Abstract (translated)
判别分类器与生成式分类器之间的比较自Efron对逻辑回归和判别分析的开创性分析以来，一直吸引着研究人员的兴趣。早期的理论工作表明，在简单的线性设置中，生成式分类器表现出更低的数据复杂度但更高的渐近误差。然而，在Transformer时代，这些权衡关系仍未得到充分探索。 我们首次全面评估了现代生成式和判别式架构——包括自回归建模、掩码语言模型、离散扩散以及用于文本分类的编码器。我们的研究揭示了经典的“两种模式”现象在不同架构和训练范式中表现出不同的特征。除了准确性之外，我们还分析了样本效率、校准性、对噪声的鲁棒性和顺序性在各种场景中的表现。 这些发现为基于现实世界约束（如延迟和数据限制）选择最合适的建模方法提供了实用指导。
URL
https://arxiv.org/abs/2506.12181
PDF
-
GeistBERT: Breathing Life into German NLP
2025-06-13 15:53:17Raphael Scheible-Schmitt, Johann FreiAbstract
Advances in transformer-based language models have highlighted the benefits of language-specific pre-training on high-quality corpora. In this context, German NLP stands to gain from updated architectures and modern datasets tailored to the linguistic characteristics of the German language. GeistBERT seeks to improve German language processing by incrementally training on a diverse corpus and optimizing model performance across various NLP tasks. It was pre-trained using fairseq with standard hyperparameters, initialized from GottBERT weights, and trained on a large-scale German corpus using Whole Word Masking (WWM). Based on the pre-trained model, we derived extended-input variants using Nyströmformer and Longformer architectures with support for sequences up to 8k tokens. While these long-context models were not evaluated on dedicated long-context benchmarks, they are included in our release. We assessed all models on NER (CoNLL 2003, GermEval 2014) and text classification (GermEval 2018 fine/coarse, 10kGNAD) using $F_1$ score and accuracy. The GeistBERT models achieved strong performance, leading all tasks among the base models and setting a new state-of-the-art (SOTA). Notably, the base models outperformed larger models in several tasks. To support the German NLP research community, we are releasing GeistBERT under the MIT license.
Abstract (translated)
基于变压器的语言模型的进展强调了在高质量语料库上进行语言特定预训练的好处。在此背景下，德语文本处理可以从更新的架构和专为德语语言特点设计的现代数据集中获益。GeistBERT旨在通过在多样化语料库上逐步训练并优化各种NLP任务中的模型性能来改进德文语言处理。它使用fairseq进行预训练，并采用标准超参数，从GottBERT权重初始化，然后使用Whole Word Masking（WWM）技术在一个大规模的德语文本语料库上进行训练。 基于这个预训练模型，我们利用Nyströmformer和Longformer架构生成了支持最长达8000个token序列的扩展输入变体。尽管这些长上下文模型并未在专门针对长上下文基准测试中评估，但它们已包含在我们的发布版本中。 我们在命名实体识别（CoNLL 2003, GermEval 2014）和文本分类（GermEval 2018细/粗粒度，10kGNAD）任务上对所有模型进行了测试，并使用$F_1$分数和准确率作为评价标准。GeistBERT模型在所有基础模型中均表现出色，在多个任务中设立了新的最佳状态（SOTA）。值得注意的是，在某些任务中，基础模型的表现优于更大的模型。 为了支持德语文本处理研究社区的发展，我们将在MIT许可证下发布GeistBERT。
URL
https://arxiv.org/abs/2506.11903
PDF
-
Flick: Few Labels Text Classification using K-Aware Intermediate Learning in Multi-Task Low-Resource Languages
2025-06-12 02:09:47Ali Almutairi, Abdullah Alsuhaibani, Shoaib Jameel, Usman Naseem, Gelareh Mohammadi, Imran RazzakAbstract
Training deep learning networks with minimal supervision has gained significant research attention due to its potential to reduce reliance on extensive labelled data. While self-training methods have proven effective in semi-supervised learning, they remain vulnerable to errors from noisy pseudo labels. Moreover, most recent approaches to the few-label classification problem are either designed for resource-rich languages such as English or involve complex cascading models that are prone to overfitting. To address the persistent challenge of few-label text classification in truly low-resource linguistic contexts, where existing methods often struggle with noisy pseudo-labels and domain adaptation, we propose Flick. Unlike prior methods that rely on generic multi-cluster pseudo-labelling or complex cascading architectures, Flick leverages the fundamental insight that distilling high-confidence pseudo-labels from a broader set of initial clusters can dramatically improve pseudo-label quality, particularly for linguistically diverse, low-resource settings. Flick introduces a novel pseudo-label refinement component, a departure from traditional pseudo-labelling strategies by identifying and leveraging top-performing pseudo-label clusters. This component specifically learns to distil highly reliable pseudo-labels from an initial broad set by focusing on single-cluster cohesion and leveraging an adaptive top-k selection mechanism. This targeted refinement process is crucial for mitigating the propagation of errors inherent in low-resource data, allowing for robust fine-tuning of pre-trained language models with only a handful of true labels. We demonstrate Flick's efficacy across 14 diverse datasets, encompassing challenging low-resource languages such as Arabic, Urdu, and Setswana, alongside English, showcasing its superior performance and adaptability.
Abstract (translated)
使用最少监督训练深度学习网络因其能够减少对大量标注数据的依赖而受到了广泛关注。虽然自训练方法在半监督学习中证明了其有效性，但它们仍然容易受到由噪声伪标签引发错误的影响。此外，大多数近期解决少量标签分类问题的方法要么是为资源丰富的语言（如英语）设计的，要么涉及可能导致过拟合的复杂级联模型。为了应对真正低资源语言环境中的少量文本分类挑战，在这种环境中现有方法往往难以处理噪声伪标签和领域适应的问题，我们提出了Flick。 不同于以往依赖于通用多聚类伪标签或复杂级联架构的方法，Flick 利用了这样一个基本见解：从一个更广泛的初始聚类集中提炼出高可信度的伪标签可以显著提高伪标签的质量，特别是在语言多样且资源有限的情况下。Flick 引入了一种新颖的伪标签精炼组件，这与传统的伪标签策略有所不同，通过识别和利用表现最佳的伪标签聚类来实现改进。该组件专门学习从一个初始宽泛集中的单个聚类中提炼出高度可靠的伪标签，并通过采用自适应Top-K选择机制来提高单一聚类内的一致性。这种针对性的精炼过程对于缓解低资源数据中错误传播的问题至关重要，使得仅使用少量真实标签即可实现预训练语言模型的稳健微调成为可能。 我们展示了Flick 在涵盖包括阿拉伯语、乌尔都语和塞茨瓦纳语等具有挑战性的低资源语言以及英语在内的14个多样化的数据集上的有效性，证明了其卓越性能及适应性。
URL
https://arxiv.org/abs/2506.10292
PDF
