Text_Classification
-
Hierarchical Attention Network for Interpretable ECG-based Heart Disease Classification
2025-03-25 13:06:06Mario Padilla Rodriguez, Mohamed NafeaAbstract
Cardiovascular disease remains one of the leading causes of mortality worldwide, underscoring the need for accurate as well as interpretable diagnostic machine learning tools. In this work, we investigate heart disease classification using electrocardiogram (ECG) data from two widely-utilized datasets: The MIT-BIH Arrhythmia and the PTB-XL datasets. We adapt a hierarchical attention network (HAN), originally developed for text classification, into an ECG-based heart-disease classification task. Our adapted HAN incorporates two attention layers that focus on ECG data segments of varying sizes. We conduct a comparative analysis between our adapted HAN and a more sophisticated state-of-the-art architecture, featuring a network with convolution, attention, and transformer layers (CAT-Net). Our empirical evaluation encompasses multiple aspects including test accuracy (quantified by 0-1 loss); model complexity (measured by the number of model parameters); and interpretability (through attention map visualization). Our adapted HAN demonstrates comparable test accuracy with significant reductions in model complexity and enhanced interpretability analysis: For the MIT-BIH dataset, our adapted HAN achieves 98.55\% test accuracy compared to 99.14\% for CAT-Net, while reducing the number of model parameters by a factor of 15.6. For the PTB-XL dataset, our adapted HAN achieves a 19.3-fold reduction in model complexity compared to CAT-Net, with only a 5\% lower test accuracy. From an interpretability perspective, the significantly simpler architecture and the hierarchical nature of our adapted HAN model facilitate a more straightforward interpretability analysis based on visualizing attention weights. Building on this advantage, we conduct an interpretability analysis of our HAN that highlights the regions of the ECG signal most relevant to the model's decisions.
Abstract (translated)
心血管疾病仍然是全球主要的死亡原因之一，这凸显了准确且易于解释的诊断机器学习工具的需求。在这项研究中，我们使用心电图（ECG）数据对心脏病进行分类，这些数据来自两个广泛使用的数据集：MIT-BIH心律失常数据集和PTB-XL数据集。我们将最初为文本分类开发的分层注意力网络（HAN）适应到基于ECG的心脏病分类任务中。我们的改进型HAN包含两个关注不同大小的ECG数据片段的关注层。 我们对改进型HAN与一种更复杂的最新架构——具有卷积、注意和变换器层的CAT-Net进行了比较分析，后者在模型复杂性和准确性方面表现出色。我们的实证评估包括多个方面：测试准确度（通过0-1损失衡量）；模型复杂性（通过模型参数的数量衡量）；以及可解释性（通过注意力图可视化）。改进型HAN展示了与CAT-Net相当的测试准确率，但显著降低了模型复杂性和提高了可解释性： - 对于MIT-BIH数据集，我们的改进型HAN达到了98.55%的测试准确度，而CAT-Net为99.14%，同时减少了15.6倍的模型参数数量。 - 在PTB-XL数据集中，与CAT-Net相比，我们的改进型HAN将模型复杂性降低了19.3倍，仅在测试准确性上低了5%。 从可解释性的角度来看，我们简化后的架构和分层特性使得基于可视化注意力权重的解释分析更加直接。在此优势的基础上，我们对HAN进行了详细的可解释性分析，突出了ECG信号中模型决策最相关的区域。
URL
https://arxiv.org/abs/2504.03703
PDF
-
Teaching LLMs for Step-Level Automatic Math Correction via Reinforcement Learning
2025-03-24 08:28:34Junsong Li, Jie Zhou, Yutao Yang, Bihao Zhan, Qianjun Pan, Yuyang Ding, Qin Chen, Jiang Bo, Xin Lin, Liang HeAbstract
Automatic math correction aims to check students' solutions to mathematical problems via artificial intelligence technologies. Most existing studies focus on judging the final answer at the problem level, while they ignore detailed feedback on each step in a math problem-solving process, which requires abilities of semantic understanding and reasoning. In this paper, we propose a reinforcement learning (RL)-based method to boost large language model (LLM) for step-level automatic math correction, named StepAMC. Particularly, we convert the step-level automatic math correction within the text classification task into an RL problem to enhance the reasoning capabilities of LLMs. Then, we design a space-constrained policy network to improve the stability of RL. Then, we introduce a fine-grained reward network to convert the binary human feedback into a continuous value. We conduct extensive experiments over two benchmark datasets and the results show that our model outperforms the eleven strong baselines.
Abstract (translated)
自动数学纠错旨在通过人工智能技术检查学生解答数学问题的过程。现有大多数研究主要集中在评判每个问题的最终答案，而忽视了对解决问题过程中每一步的具体反馈，这需要具备语义理解和推理的能力。本文提出了一种基于强化学习（RL）的方法来增强大语言模型（LLM），用于步骤级别的自动数学纠错，并将其命名为StepAMC。特别地，我们将文本分类任务中的步骤级自动数学纠错问题转化为一个RL问题，以提升LLMs的推理能力。然后设计了一个空间受限的策略网络，以提高RL的稳定性。接着引入了一种细粒度奖励网络，将二元人类反馈转换为连续值。我们在两个基准数据集上进行了广泛的实验，并且结果显示我们的模型优于11个强大的基线模型。
URL
https://arxiv.org/abs/2503.18432
PDF
-
Sun-Shine: A Large Language Model for Tibetan Culture
2025-03-24 02:17:41Cheng Huang, Fan Gao, Nyima Tashi, Yutong Liu, Xiangxiang Wang, Thupten Tsering, Ban Ma-bao, Renzeg Duojie, Gadeng Luosang, Rinchen Dongrub, Dorje Tashi, Xiao Feng, Yongbin YuAbstract
Tibetan, a minority language in China, features a highly intricate grammatical structure, characterized by four verb tenses and a tense system with frequent irregularities, contributing to its extensive inflectional diversity. Recently, advances in Large Language Models (LLMs) have transformed the paradigm in many domains. Despite the success in other fields, current LLMs often fall short in catering to the needs of domain experts like Tibetans, and the potential of LLMs for Tibetan culture is under-explored. The intrinsic reasons are the immense and intricate nature of Tibetan culture as well as the necessity for higher granularity and richness in knowledge. Simultaneously, the complexity and uniqueness of its grammatical structure, coupled with its status as a minority ethnic language, contribute to data scarcity, which remains a fundamental challenge. To alleviate these issues, we introduce Llama-Sunshine (Sun-Shine), the first large language model for Tibetan culture, which is expert in various Tibetan language processing tasks. Sun-Shine incorporates state-of-the-art model architectures optimized for Tibetan's linguistic features. We also propose TIB-STC, a comprehensive dataset comprising diverse Tibetan texts such as literature, religious scripts, news, and conversational data, which is also the first large-scale dataset for Tibetan culture. Though comprehensive experiments, Sun-Shine not only demonstrates a higher level of knowledge expertise for Tibetan culture but also gains preliminary embodied intelligence capabilities in Tibetan language processing tasks, like language modeling, text classification, machine translation, and syntactic analysis. Moreover, it excels in low-resource scenarios, showcasing strong generalization capabilities.
Abstract (translated)
翻译成中文如下： 藏语，作为中国的少数民族语言之一，具有极其复杂的语法结构，包括四个时态和一个经常出现不规则变化的时制系统，这导致了其广泛的词形变化多样性。近年来，大型语言模型（LLMs）在许多领域中改变了范式。尽管在其他领域的应用取得了成功，当前的LLM通常无法满足像藏族这样的专业人员的需求，并且这些模型在藏文化的潜力方面尚未得到充分探索。内在原因是藏文化宏大的复杂性和知识上的细致程度和丰富度的要求。同时，其语法结构的独特复杂性以及作为少数民族语言的地位导致了数据稀缺的问题，这仍然是一个根本性的挑战。为了缓解这些问题，我们推出了Llama-Sunshine（Sun-Shine），这是第一个专门针对藏文化的大型语言模型，擅长处理各种藏语任务。Sun-Shine融合了最先进的模型架构，并针对藏语的语言特征进行了优化。此外，我们还提出了TIB-STC，这是一个包含多种藏文文本的综合性数据集，例如文学作品、宗教文献、新闻和对话数据，也是第一个大规模的藏文化数据集。通过全面的实验，Sun-Shine不仅展示了对藏文化的更高知识专业度，并且在语言建模、文本分类、机器翻译及句法分析等藏语处理任务中获得了初步的实体智能能力。此外，在资源匮乏的情况下，它也表现出强大的泛化能力。
URL
https://arxiv.org/abs/2503.18288
PDF
-
Corrective In-Context Learning: Evaluating Self-Correction in Large Language Models
2025-03-20 10:39:39Mario Sanz-Guerrero, Katharina von der WenseAbstract
In-context learning (ICL) has transformed the use of large language models (LLMs) for NLP tasks, enabling few-shot learning by conditioning on labeled examples without finetuning. Despite its effectiveness, ICL is prone to errors, especially for challenging examples. With the goal of improving the performance of ICL, we propose corrective in-context learning (CICL), an approach that incorporates a model's incorrect predictions alongside ground truth corrections into the prompt, aiming to enhance classification accuracy through self-correction. However, contrary to our hypothesis, extensive experiments on text classification tasks demonstrate that CICL consistently underperforms standard ICL, with performance degrading as the proportion of corrections in the prompt increases. Our findings indicate that CICL introduces confusion by disrupting the model's task understanding, rather than refining its predictions. Additionally, we observe that presenting harder examples in standard ICL does not improve performance, suggesting that example difficulty alone may not be a reliable criterion for effective selection. By presenting these negative results, we provide important insights into the limitations of self-corrective mechanisms in LLMs and offer directions for future research.
Abstract (translated)
基于上下文的学习（ICL）已经改变了大型语言模型（LLMs）在自然语言处理（NLP）任务中的使用方式，通过利用标注示例进行少量样本学习而不需微调。尽管ICL非常有效，但它容易出错，尤其是在面对难题时。为了改进ICL的性能，我们提出了校正上下文学习（CICL），这是一种将模型错误预测和真实纠正结果整合到提示中的方法，旨在通过自我修正来提高分类精度。然而，与我们的假设相反，在文本分类任务上的大量实验表明，CICL的表现始终逊于标准ICL，并且随着提示中纠错比例的增加，性能还会下降。我们的研究发现指出，CICL引入了混淆，破坏了模型的任务理解能力，而不是改进其预测结果。此外，我们还观察到在标准ICL中呈现更难的例子并不能提高表现，这表明难度本身可能不是有效选择示例的一个可靠标准。通过展示这些负面结果，我们为LLMs自我修正机制的局限性提供了重要见解，并为进一步研究指明了方向。
URL
https://arxiv.org/abs/2503.16022
PDF
-
Dynamic Bi-Elman Attention Networks : Dual-Directional Context-Aware Representation Learning for Enhanced Text Classification
2025-03-19 17:45:13ZhengLin Lai, MengYao Liao, Dong XuAbstract
Text classification, a fundamental task in natural language processing (NLP), aims to categorize textual data into predefined labels. Traditional methods struggled with complex linguistic structures and semantic dependencies. The advent of deep learning, particularly recurrent neural networks (RNNs) and Transformer-based models, has significantly advanced the field by enabling nuanced feature extraction and context-aware predictions. Despite improvements, existing models exhibit limitations in balancing interpretability, computational efficiency, and long-range contextual understanding. This paper proposes the Dynamic Bidirectional Elman with Attention Network (DBEAN), which integrates bidirectional temporal modelling with self-attention mechanisms. DBEAN dynamically assigns weights to critical segments of input, improving contextual representation while maintaining computational efficiency.
Abstract (translated)
文本分类是自然语言处理（NLP）中的一个基本任务，旨在将文本数据归类到预定义的标签中。传统方法在处理复杂的语言结构和语义依赖方面遇到了困难。深度学习技术的发展，尤其是循环神经网络（RNNs）和基于Transformer的模型的应用，极大地推动了该领域的发展，使得细微特征提取和上下文感知预测成为可能。然而，现有的模型在可解释性、计算效率以及长程上下文理解之间存在难以平衡的问题。 本文提出了一种新的方法：动态双向Elman注意力网络（DBEAN），它将双向时间建模与自注意力机制相结合。DBEAN能够根据输入文本的重要部分动态分配权重，在提升上下文表示能力的同时保持较高的计算效率。
URL
https://arxiv.org/abs/2503.15469
PDF
-
Comparing Llama3 and DeepSeekR1 on Biomedical Text Classification Tasks
2025-03-19 12:51:52Yuting Guo, Abeed SarkerAbstract
This study compares the performance of two open-source large language models (LLMs)-Llama3-70B and DeepSeekR1-distill-Llama3-70B-on six biomedical text classification tasks. Four tasks involve data from social media, while two tasks focus on clinical notes from electronic health records, and all experiments were performed in zero-shot settings. Performance metrics, including precision, recall, and F1 scores, were measured for each task, along with their 95% confidence intervals. Results demonstrated that DeepSeekR1-distill-Llama3-70B generally performs better in terms of precision on most tasks, with mixed results on recall. While the zero-shot LLMs demonstrated high F1 scores for some tasks, they grossly underperformed on others, for data from both sources. The findings suggest that model selection should be guided by the specific requirements of the health-related text classification tasks, particularly when considering the precision-recall trade-offs, and that, in the presence of annotated data, supervised classification approaches may be more reliable than zero-shot LLMs.
Abstract (translated)
这项研究比较了两个开源大型语言模型（LLMs）——Llama3-70B 和 DeepSeekR1-distill-Llama3-70B 在六个生物医学文本分类任务中的性能。其中四个任务涉及社交媒体数据，而另外两个任务则侧重于电子健康记录中的临床笔记，并且所有实验均在零样本设置下进行。每个任务的性能指标包括准确率、召回率和F1值，以及95%置信区间。 结果显示，在大多数任务中，DeepSeekR1-distill-Llama3-70B 在准确率方面表现更好，而对召回率的结果则较为混合。尽管零样本LLMs在某些任务上表现出较高的F1分数，但在另一些任务上的表现明显不足，无论是社交媒体数据还是电子健康记录中的数据。 研究结果表明，在选择模型时应考虑特定的健康相关文本分类任务的具体需求，特别是在权衡准确率和召回率方面。此外，当存在标注数据时，监督分类方法可能比零样本LLMs更为可靠。
URL
https://arxiv.org/abs/2503.15169
PDF
-
A Comprehensive Survey on Architectural Advances in Deep CNNs: Challenges, Applications, and Emerging Research Directions
2025-03-19 08:41:06Saddam Hussain Khan (Artificial Intelligence Lab, Department of Computer Systems Engineering, University of Engineering,Applied Sciences), Rashid Iqbal (Artificial Intelligence Lab, Department of Computer Systems Engineering, University of Engineering,Applied Sciences)arXiv_AI Speech_Recognition CNN Recognition Weakly_Supervised Action_Recognition Detection Object_Detection Deep_Learning Review Survey Classification Face Face_Recognition Attention Text_Classification Transfer_Learning Pose_Estimation Inference Prediction Language_Model Sentiment Transformer Pose Action Few-Shot Zero-Shot Medical 3D Enhancement SpeechAbstract
Deep Convolutional Neural Networks (CNNs) have significantly advanced deep learning, driving breakthroughs in computer vision, natural language processing, medical diagnosis, object detection, and speech recognition. Architectural innovations including 1D, 2D, and 3D convolutional models, dilated and grouped convolutions, depthwise separable convolutions, and attention mechanisms address domain-specific challenges and enhance feature representation and computational efficiency. Structural refinements such as spatial-channel exploitation, multi-path design, and feature-map enhancement contribute to robust hierarchical feature extraction and improved generalization, particularly through transfer learning. Efficient preprocessing strategies, including Fourier transforms, structured transforms, low-precision computation, and weight compression, optimize inference speed and facilitate deployment in resource-constrained environments. This survey presents a unified taxonomy that classifies CNN architectures based on spatial exploitation, multi-path structures, depth, width, dimensionality expansion, channel boosting, and attention mechanisms. It systematically reviews CNN applications in face recognition, pose estimation, action recognition, text classification, statistical language modeling, disease diagnosis, radiological analysis, cryptocurrency sentiment prediction, 1D data processing, video analysis, and speech recognition. In addition to consolidating architectural advancements, the review highlights emerging learning paradigms such as few-shot, zero-shot, weakly supervised, federated learning frameworks and future research directions include hybrid CNN-transformer models, vision-language integration, generative learning, etc. This review provides a comprehensive perspective on CNN's evolution from 2015 to 2025, outlining key innovations, challenges, and opportunities.
Abstract (translated)
深度卷积神经网络（CNNs）在深度学习领域取得了重大进展，推动了计算机视觉、自然语言处理、医学诊断、目标检测和语音识别等领域的突破。包括一维、二维及三维卷积模型、扩张卷积与分组卷积、逐深可分离卷积以及注意力机制在内的架构创新解决了特定领域的挑战，并增强了特征表示能力及计算效率。空间-通道利用、多路径设计及特征图增强等结构改进有助于稳健的层级特征提取和泛化性能提升，特别是在迁移学习中表现尤为突出。有效的预处理策略，如傅里叶变换、结构转换、低精度运算及权重压缩，则优化了推理速度并支持在资源受限环境中部署应用。 本文综述提出了一种统一分类体系，根据空间利用、多路径架构、深度、宽度、维度扩展、通道增强和注意力机制对CNN架构进行分类。系统性地回顾了CNN在人脸识别、姿态估计、动作识别、文本分类、统计语言建模、疾病诊断、放射学分析、加密货币情感预测、一维数据处理、视频分析及语音识别等领域的应用。除了总结结构上的进步，还强调了新兴的学习范式如小样本学习（few-shot learning）、零样本学习（zero-shot learning）、弱监督学习和联邦学习框架，并展望了未来研究方向包括混合CNN-Transformer模型、视觉语言整合以及生成性学习等。 本文提供了2015年至2025年间CNN进化历程的全面视角，概述了关键创新、挑战及机遇。
URL
https://arxiv.org/abs/2503.16546
PDF
-
Micro Text Classification Based on Balanced Positive-Unlabeled Learning
2025-03-17 07:42:27Lin-Han Jia, Lan-Zhe Guo, Zhi Zhou, Si-Ye Han, Zi-Wen Li, Yu-Feng LiAbstract
In real-world text classification tasks, negative texts often contain a minimal proportion of negative content, which is especially problematic in areas like text quality control, legal risk screening, and sensitive information interception. This challenge manifests at two levels: at the macro level, distinguishing negative texts is difficult due to the high similarity between coarse-grained positive and negative samples; at the micro level, the issue stems from extreme class imbalance and a lack of fine-grained labels. To address these challenges, we propose transforming the coarse-grained positive-negative (PN) classification task into an imbalanced fine-grained positive-unlabeled (PU) classification problem, supported by theoretical analysis. We introduce a novel framework, Balanced Fine-Grained Positive-Unlabeled (BFGPU) learning, which features a unique PU learning loss function that optimizes macro-level performance amidst severe imbalance at the micro level. The framework's performance is further boosted by rebalanced pseudo-labeling and threshold adjustment. Extensive experiments on both public and real-world datasets demonstrate the effectiveness of BFGPU, which outperforms other methods, even in extreme scenarios where both macro and micro levels are highly imbalanced.
Abstract (translated)
在实际文本分类任务中，负面文本往往只包含少量的负面内容，在诸如文本质量控制、法律风险筛查和敏感信息拦截等领域的应用尤其成问题。这一挑战体现在两个层面：宏观层面上，由于粗粒度正负样本之间的高度相似性，区分负面文本变得困难；微观层面上，则是由极端类别不平衡以及缺乏细粒度标签所导致的问题。为解决这些挑战，我们提出将粗粒度的正负（PN）分类任务转换为不平衡的细粒度正未标记（PU）分类问题，并通过理论分析加以支持。为此，我们引入了一种全新的框架——平衡细粒度正未标记（BFGPU）学习，该框架采用一种独特的PU学习损失函数，在微观层面上面对极端不平衡的情况下优化宏观层面的表现。此外，框架的性能还得到了重置伪标签和阈值调整技术的支持。在公共数据集及真实世界数据集上的广泛实验显示，BFGPU方法表现出色，并且即使是在极度失衡（无论是宏观层面还是微观层面）的情境下，它依然优于其他所有方法。
URL
https://arxiv.org/abs/2503.13562
PDF
-
Wikipedia is Not a Dictionary, Delete! Text Classification as a Proxy for Analysing Wiki Deletion Discussions
2025-03-13 12:07:35Hsuvas Borkakoty, Luis Espinosa-AnkeAbstract
Automated content moderation for collaborative knowledge hubs like Wikipedia or Wikidata is an important yet challenging task due to multiple factors. In this paper, we construct a database of discussions happening around articles marked for deletion in several Wikis and in three languages, which we then use to evaluate a range of LMs on different tasks (from predicting the outcome of the discussion to identifying the implicit policy an individual comment might be pointing to). Our results reveal, among others, that discussions leading to deletion are easier to predict, and that, surprisingly, self-produced tags (keep, delete or redirect) don't always help guiding the classifiers, presumably because of users' hesitation or deliberation within comments.
Abstract (translated)
针对维基百科或Wikidata等协作知识中心的自动化内容审核是一项重要但具有挑战性的任务，原因众多。本文中，我们构建了一个数据库，涵盖了多个维基站点上标记为待删除文章的相关讨论，并涉及三种语言的数据。然后，我们使用这些数据来评估一系列大型语言模型（LMs）在不同任务上的表现（从预测讨论的结果到识别个人评论可能指向的隐含政策）。我们的研究结果表明，例如，导致内容被删除的讨论更容易被预测；此外，令人意外的是，用户自动生成的标签（保留、删除或重定向）并不总是有助于引导分类器做出判断，这可能是由于用户在评论中表现出犹豫不决或深思熟虑所致。
URL
https://arxiv.org/abs/2503.10294
PDF
-
Cognitive-Mental-LLM: Leveraging Reasoning in Large Language Models for Mental Health Prediction via Online Text
2025-03-13 06:42:37Avinash Patil, Amardeep Kour GedhuAbstract
Large Language Models (LLMs) have demonstrated potential in predicting mental health outcomes from online text, yet traditional classification methods often lack interpretability and robustness. This study evaluates structured reasoning techniques-Chain-of-Thought (CoT), Self-Consistency (SC-CoT), and Tree-of-Thought (ToT)-to improve classification accuracy across multiple mental health datasets sourced from Reddit. We analyze reasoning-driven prompting strategies, including Zero-shot CoT and Few-shot CoT, using key performance metrics such as Balanced Accuracy, F1 score, and Sensitivity/Specificity. Our findings indicate that reasoning-enhanced techniques improve classification performance over direct prediction, particularly in complex cases. Compared to baselines such as Zero Shot non-CoT Prompting, and fine-tuned pre-trained transformers such as BERT and Mental-RoBerta, and fine-tuned Open Source LLMs such as Mental Alpaca and Mental-Flan-T5, reasoning-driven LLMs yield notable gains on datasets like Dreaddit (+0.52\% over M-LLM, +0.82\% over BERT) and SDCNL (+4.67\% over M-LLM, +2.17\% over BERT). However, performance declines in Depression Severity, and CSSRS predictions suggest dataset-specific limitations, likely due to our using a more extensive test set. Among prompting strategies, Few-shot CoT consistently outperforms others, reinforcing the effectiveness of reasoning-driven LLMs. Nonetheless, dataset variability highlights challenges in model reliability and interpretability. This study provides a comprehensive benchmark of reasoning-based LLM techniques for mental health text classification. It offers insights into their potential for scalable clinical applications while identifying key challenges for future improvements.
Abstract (translated)
大型语言模型（LLMs）在从在线文本预测心理健康结果方面展示了潜力，但传统的分类方法往往缺乏可解释性和鲁棒性。本研究评估了结构化推理技术——链式思维（CoT）、自我一致性（SC-CoT）和思想树（ToT），以提高来自Reddit的多个心理健康数据集上的分类准确性。我们分析了包括零样本链式思维和少量样本链式思维在内的基于推理的提示策略，并使用平衡准确率、F1分数以及敏感度/特异度等关键性能指标进行评估。我们的研究发现表明，增强推理的技术比直接预测在复杂案例中提高了分类性能。与基线模型（如零样本非CoT提示和预训练转换器BERT及Mental-RoBerta）相比，以及与开源LLMs（如Mental Alpaca和Mental-Flan-T5）微调后的结果相比，基于推理的LLMs在Dreaddit数据集上提高了0.52%（相对于M-LLM）和0.82%（相对于BERT），在SDCNL数据集上分别提升了4.67%（相对于M-LLM）和2.17%（相对于BERT）。然而，在抑郁症严重程度以及CSSRS预测中的性能下降表明了特定于数据集的限制，这可能是由于使用了更广泛的数据测试集。在提示策略中，少量样本链式思维一贯优于其他方法，进一步验证了基于推理的LLMs的有效性。尽管如此，数据集的变化突显了模型可靠性和可解释性的挑战。 本研究为基于推理的LLM技术在心理健康文本分类上的全面基准提供了依据，并为其在未来临床应用中的潜力和面临的关键挑战提供了见解。
URL
https://arxiv.org/abs/2503.10095
PDF
-
Ordered Semantically Diverse Sampling for Textual Data
2025-03-12 06:38:57Ashish Tiwari, Mukul Singh, Ananya Singha, Arjun RadhakrishnaAbstract
The goal of diversity sampling is to select a representative subset of data in a way that maximizes information contained in the subset while keeping its cardinality small. We introduce the ordered diverse sampling problem based on a new metric that measures the diversity in an ordered list of samples. We present a novel approach for generating ordered diverse samples for textual data that uses principal components on the embedding vectors. The proposed approach is simple and compared with existing approaches using the new metric. We transform standard text classification benchmarks into benchmarks for ordered diverse sampling. Our empirical evaluation shows that prevailing approaches perform 6% to 61% worse than our method while also being more time inefficient. Ablation studies show how the parts of the new approach contribute to the overall metrics.
Abstract (translated)
多样性采样的目标是以最大化子集中包含的信息量的方式，选择具有代表性的数据子集，同时保持其基数较小。我们基于一个新的度量标准引入了有序多样本抽样问题，该度量标准用于衡量样本有序列表中的多样性。我们提出了一种新颖的方法，通过在嵌入向量上使用主成分分析来生成文本数据的有序多样化样本。所提出的这种方法是简单的，并且与现有方法相比，在新的度量标准下表现更优。我们将传统的文本分类基准转换为有序多样本抽样的基准测试。我们的实证评估表明，现有的方法比我们的方法性能低6%到61%，并且在时间效率上也较低效。消融研究表明了新方法的各个部分如何贡献于整体指标的表现。
URL
https://arxiv.org/abs/2503.10698
PDF
-
LabelCoRank: Revolutionizing Long Tail Multi-Label Classification with Co-Occurrence Reranking
2025-03-11 01:52:39Yan Yan, Junyuan Liu, Bo-Wen ZhangAbstract
Motivation: Despite recent advancements in semantic representation driven by pre-trained and large-scale language models, addressing long tail challenges in multi-label text classification remains a significant issue. Long tail challenges have persistently posed difficulties in accurately classifying less frequent labels. Current approaches often focus on improving text semantics while neglecting the crucial role of label relationships. Results: This paper introduces LabelCoRank, a novel approach inspired by ranking principles. LabelCoRank leverages label co-occurrence relationships to refine initial label classifications through a dual-stage reranking process. The first stage uses initial classification results to form a preliminary ranking. In the second stage, a label co-occurrence matrix is utilized to rerank the preliminary results, enhancing the accuracy and relevance of the final classifications. By integrating the reranked label representations as additional text features, LabelCoRank effectively mitigates long tail issues in multi-labeltext classification. Experimental evaluations on popular datasets including MAG-CS, PubMed, and AAPD demonstrate the effectiveness and robustness of LabelCoRank.
Abstract (translated)
动机：尽管由预训练和大规模语言模型驱动的语义表示取得了近期进展，但在多标签文本分类中的长尾挑战仍然是一个重大问题。长期存在的难题在于准确分类较少出现的标签上。目前的方法通常侧重于改进文本语义，而忽视了标签间关系的关键作用。 结果：本文提出了一种新颖的方法LabelCoRank，该方法受排名原则启发。LabelCoRank通过利用标签共现关系，在双重重排序过程中优化初始标签分类。第一阶段使用初步的分类结果来形成一个初级排名；在第二阶段中，则使用标签共现矩阵对初步结果进行重新排序，以提高最终分类的准确性和相关性。通过将重排序后的标签表示集成到文本特征中，LabelCoRank能够有效缓解多标签文本分类中的长尾问题。实验评估显示，在包括MAG-CS、PubMed和AAPD在内的流行数据集上，LabelCoRank表现出其有效的性能和鲁棒性。
URL
https://arxiv.org/abs/2503.07968
PDF
-
Fair Text Classification via Transferable Representations
2025-03-10 16:52:45Thibaud Leteno, Michael Perrot, Charlotte Laclau, Antoine Gourru, Christophe GravierAbstract
Group fairness is a central research topic in text classification, where reaching fair treatment between sensitive groups (e.g., women and men) remains an open challenge. We propose an approach that extends the use of the Wasserstein Dependency Measure for learning unbiased neural text classifiers. Given the challenge of distinguishing fair from unfair information in a text encoder, we draw inspiration from adversarial training by inducing independence between representations learned for the target label and those for a sensitive attribute. We further show that Domain Adaptation can be efficiently leveraged to remove the need for access to the sensitive attributes in the dataset we cure. We provide both theoretical and empirical evidence that our approach is well-founded.
Abstract (translated)
群体公平性是文本分类研究中的一个核心课题，其中在敏感群体（例如女性和男性）之间实现公平待遇仍然是一个未解决的挑战。我们提出了一种方法，该方法扩展了Wasserstein依赖度量的应用，以学习无偏见的神经文本分类器。面对区分文本编码器中公正信息与不公正信息的难题，我们借鉴对抗训练的思想，通过诱导目标标签和敏感属性之间表示的独立性来实现这一目的。此外，我们进一步表明，领域适应技术可以有效地利用，从而在处理数据集时无需访问敏感属性。我们提供了理论和实证证据，证明我们的方法是合理且有效的。
URL
https://arxiv.org/abs/2503.07691
PDF
-
Is My Text in Your AI Model? Gradient-based Membership Inference Test applied to LLMs
2025-03-10 14:32:56Gonzalo Mancera, Daniel de Alcala, Julian Fierrez, Ruben Tolosana, Aythami MoralesAbstract
This work adapts and studies the gradient-based Membership Inference Test (gMINT) to the classification of text based on LLMs. MINT is a general approach intended to determine if given data was used for training machine learning models, and this work focuses on its application to the domain of Natural Language Processing. Using gradient-based analysis, the MINT model identifies whether particular data samples were included during the language model training phase, addressing growing concerns about data privacy in machine learning. The method was evaluated in seven Transformer-based models and six datasets comprising over 2.5 million sentences, focusing on text classification tasks. Experimental results demonstrate MINTs robustness, achieving AUC scores between 85% and 99%, depending on data size and model architecture. These findings highlight MINTs potential as a scalable and reliable tool for auditing machine learning models, ensuring transparency, safeguarding sensitive data, and fostering ethical compliance in the deployment of AI/NLP technologies.
Abstract (translated)
这项工作将基于梯度的成员推断测试（gMINT）应用于基于大型语言模型（LLMs）的文本分类，并对其进行了研究。MINT是一种通用方法，旨在确定给定的数据是否用于训练机器学习模型，而本工作则重点关注其在自然语言处理领域的应用。通过基于梯度的分析，MINT模型能够识别特定数据样本是否包含于语言模型的训练阶段中，从而应对日益增长的关于机器学习中的数据隐私问题的关注。 该方法在七个基于Transformer的模型和六个包含超过250万句子的数据集上进行了评估，重点在于文本分类任务。实验结果表明，MINT具有较强的鲁棒性，在不同数据规模和模型架构下达到了85%到99%之间的AUC评分。这些发现凸显了MINT作为一种可扩展且可靠的工具在机器学习模型审计中的潜力，确保透明度、保护敏感信息，并促进AI/NLP技术部署过程中的伦理合规性。
URL
https://arxiv.org/abs/2503.07384
PDF
-
Evaluation of the Automated Labeling Method for Taxonomic Nomenclature Through Prompt-Optimized Large Language Model
2025-03-08 23:11:43Keito Inoshita, Kota Nojiri, Haruto Sugeno, Takumi TagaAbstract
Scientific names of organisms consist of a genus name and a species epithet, with the latter often reflecting aspects such as morphology, ecology, distribution, and cultural background. Traditionally, researchers have manually labeled species names by carefully examining taxonomic descriptions, a process that demands substantial time and effort when dealing with large datasets. This study evaluates the feasibility of automatic species name labeling using large language model (LLM) by leveraging their text classification and semantic extraction capabilities. Using the spider name dataset compiled by Mammola et al., we compared LLM-based labeling results-enhanced through prompt engineering-with human annotations. The results indicate that LLM-based classification achieved high accuracy in Morphology, Geography, and People categories. However, classification accuracy was lower in Ecology & Behavior and Modern & Past Culture, revealing challenges in interpreting animal behavior and cultural contexts. Future research will focus on improving accuracy through optimized few-shot learning and retrieval-augmented generation techniques, while also expanding the applicability of LLM-based labeling to diverse biological taxa.
Abstract (translated)
生物的学名由属名和种加词组成，后者常反映形态、生态、分布及文化背景等方面的信息。传统上，研究人员通过仔细研究分类描述来手动标注物种名称，处理大规模数据集时需要耗费大量时间和精力。本研究评估了利用大型语言模型（LLM）自动标注物种名称的可行性，这些模型能够利用其文本分类和语义提取能力。我们使用Mammola等人编纂的蜘蛛名称数据集，将基于LLM的标签结果与人工注释进行了比较，并通过提示工程优化了LLM的表现。结果显示，在形态、地理及人类类别中，基于LLM的分类达到了高准确性；然而在生态行为和现代及历史文化类别中的分类准确性较低，显示出了解释动物行为和文化背景方面的挑战。未来的研究将致力于通过优化少样本学习和检索增强生成技术来提高准确度，并扩大基于LLM标签的应用范围至各种生物类群。
URL
https://arxiv.org/abs/2503.10662
PDF
-
Breaking Free from MMI: A New Frontier in Rationalization by Probing Input Utilization
2025-03-08 13:08:46Wei Liu, Zhiying Deng, Zhongyu Niu, Jun Wang, Haozhao Wang, Zhigang Zeng, Ruixuan LiAbstract
Extracting a small subset of crucial rationales from the full input is a key problem in explainability research. The most widely used fundamental criterion for rationale extraction is the maximum mutual information (MMI) criterion. In this paper, we first demonstrate that MMI suffers from diminishing marginal returns. Once part of the rationale has been identified, finding the remaining portions contributes only marginally to increasing the mutual information, making it difficult to use MMI to locate the rest. In contrast to MMI that aims to reproduce the prediction, we seek to identify the parts of the input that the network can actually utilize. This is achieved by comparing how different rationale candidates match the capability space of the weight matrix. The weight matrix of a neural network is typically low-rank, meaning that the linear combinations of its column vectors can only cover part of the directions in a high-dimensional space (high-dimension: the dimensions of an input vector). If an input is fully utilized by the network, {it generally matches these directions (e.g., a portion of a hypersphere), resulting in a representation with a high norm. Conversely, if an input primarily falls outside (orthogonal to) these directions}, its representation norm will approach zero, behaving like noise that the network cannot effectively utilize. Building on this, we propose using the norms of rationale candidates as an alternative objective to MMI. Through experiments on four text classification datasets and one graph classification dataset using three network architectures (GRUs, BERT, and GCN), we show that our method outperforms MMI and its improved variants in identifying better rationales. We also compare our method with a representative LLM (llama-3.1-8b-instruct) and find that our simple method gets comparable results to it and can sometimes even outperform it.
Abstract (translated)
从全文输入中提取关键理由的子集是可解释性研究中的一个核心问题。目前最广泛使用的基本标准是最大互信息（MMI）准则。在本文中，我们首先展示了MMI存在边际收益递减的问题：一旦部分原因被识别出来，找到剩余的部分只能微弱地增加互信息值，这使得利用MMI来定位其余部分变得困难。 与MMI旨在重现预测不同，我们的目标是识别网络实际可以使用的输入部分。通过比较不同的理由候选者如何匹配权重矩阵的能力空间实现这一目标。神经网络的权重矩阵通常是低秩的，这意味着其列向量的线性组合只能覆盖高维空间中的一部分方向（例如：输入向量维度）。如果一个输入被网络完全利用，则它通常与这些方向相匹配（比如超球体的一小部分），从而产生具有高范数的表示。相反地，如果一个输入主要位于（垂直于）这些方向之外，其表示范数将接近零，表现出像噪音一样无法有效利用的状态。 基于此观察，我们提出使用理由候选者的范数作为MMI的一个替代目标函数。通过在四个文本分类数据集和一个图分类数据集上进行实验，并采用三种网络架构（GRUs、BERT 和 GCN），我们展示了我们的方法在识别更好的理由方面优于MMI及其改进变体。 此外，我们将我们的方法与一个代表性的大语言模型（llama-3.1-8b-instruct）进行了比较，发现我们的简单方法能够获得与其相当的结果，并且有时甚至可以超越它。
URL
https://arxiv.org/abs/2503.06202
PDF
-
Disrupting Model Merging: A Parameter-Level Defense Without Sacrificing Accuracy
2025-03-08 06:08:47Wei Junhao, Yu Zhe, Sakuma JunAbstract
Model merging is a technique that combines multiple finetuned models into a single model without additional training, allowing a free-rider to cheaply inherit specialized capabilities. This study investigates methodologies to suppress unwanted model merging by free-riders. Existing methods such as model watermarking or fingerprinting can only detect merging in hindsight. In contrast, we propose a first proactive defense against model merging. Specifically, our defense method modifies the model parameters so that the model is disrupted if the model is merged with any other model, while its functionality is kept unchanged if not merged with others. Our approach consists of two modules, rearranging MLP parameters and scaling attention heads, which push the model out of the shared basin in parameter space, causing the merging performance with other models to degrade significantly. We conduct extensive experiments on image classification, image generation, and text classification to demonstrate that our defense severely disrupts merging while retaining the functionality of the post-protect model. Moreover, we analyze potential adaptive attacks and further propose a dropout-based pruning to improve our proposal's robustness.
Abstract (translated)
模型合并是一种技术，它将多个微调后的模型组合成一个单一的模型而不进行额外训练，从而使搭便车者能够廉价地继承专门的能力。这项研究调查了抑制搭便车者不希望看到的模型合并的方法。现有的方法如模型水印或指纹只能在事后检测到合并的情况。相比之下，我们提出了一种主动防御模型合并的方法。具体来说，我们的防御方法修改了模型参数，使得如果该模型与其他模型合并的话，其功能将被破坏；而如果没有与其他模型合并，则其功能保持不变。 我们的方法由两个模块组成：重新排列多层感知器（MLP）的参数和调整注意力头的权重，这使模型在参数空间中远离共享区域，导致与其它模型合并时性能显著下降。我们在图像分类、图像生成和文本分类上进行了广泛的实验，证明了我们的防御技术严重阻碍了模型合并的同时保留了保护后的模型的功能性。 此外，我们还分析了可能的适应性攻击，并进一步提出了基于dropout的剪枝方法以提高我们提议方案的鲁棒性。
URL
https://arxiv.org/abs/2503.07661
PDF
-
Similarity-Based Domain Adaptation with LLMs
2025-03-07 09:51:07Jie He, Wendi Zhou, Xiang Lorraine Li, Jeff Z. PanAbstract
Unsupervised domain adaptation leverages abundant labeled data from various source domains to generalize onto unlabeled target data. Prior research has primarily focused on learning domain-invariant features across the source and target domains. However, these methods often require training a model using source domain data, which is time-consuming and can limit model usage for applications with different source data. This paper introduces a simple framework that utilizes the impressive generalization capabilities of Large Language Models (LLMs) for target data annotation without the need of source model training, followed by a novel similarity-based knowledge distillation loss. Our extensive experiments on cross-domain text classification reveal that our framework achieves impressive performance, specifically, 2.44\% accuracy improvement when compared to the SOTA method.
Abstract (translated)
无监督领域自适应利用来自多个源领域的大量标记数据来推广到未标记的目标数据。先前的研究主要集中在学习源域和目标域之间的领域不变特征上。然而，这些方法通常需要使用源域数据训练模型，这既耗时又限制了在不同源数据应用中的模型使用。本文介绍了一个简单的框架，该框架利用大型语言模型（LLM）出色的泛化能力为目标数据进行标注，并且无需对源模型进行训练，随后提出了一种基于相似性的知识蒸馏损失函数。我们在跨域文本分类的广泛实验中发现，我们的框架实现了令人印象深刻的性能，在与最先进方法相比时，准确率提高了2.44%。
URL
https://arxiv.org/abs/2503.05281
PDF
-
Ensemble Debiasing Across Class and Sample Levels for Fairer Prompting Accuracy
2025-03-07 05:34:31Ruixi Lin, Ziqiao Wang, Yang YouAbstract
Language models are strong few-shot learners and achieve good overall accuracy in text classification tasks, masking the fact that their results suffer from great class accuracy imbalance. We believe that the pursuit of overall accuracy should not come from enriching the strong classes, but from raising up the weak ones. To address the imbalance, we propose a post-hoc nonlinear integer programming based debiasing method that ensembles weight correction and membership correction to enable flexible rectifications of class probabilities at both class and sample levels, enhancing the performance of LLMs directly from their outputs. Evaluations with Llama-2-13B on seven text classification benchmarks show that our approach achieves state-of-the-art overall accuracy gains with balanced class accuracies. The resulted probability correction scheme demonstrates that sample-level corrections are necessary to elevate weak classes. In addition, due to effectively correcting weak classes, our method also brings significant performance gains to Llama-2-70B, especially on a biomedical domain task, demonstrating its effectiveness across both small and large model variants.
Abstract (translated)
语言模型在少量样本学习任务中表现出色，并且在文本分类任务中的整体准确性很高，但这掩盖了一个事实：它们的结果存在严重的类别准确率不平衡。我们认为，不应通过增强强势类别来提高整体准确度，而应提升弱势类别的表现。为了解决这种不平衡问题，我们提出了一种基于后处理的非线性整数规划去偏方法，该方法结合了权重修正和成员身份修正，能够在分类级别和样本级别的灵活纠正类别概率，从而直接从语言模型输出中增强其性能。 通过使用Llama-2-13B在七个文本分类基准数据集上的评估显示，我们的方法实现了最先进的整体准确度提升，并且各类别的准确性更加均衡。所提出的概率修正方案表明，在提升弱势类别时，样本级别的修正至关重要。此外，由于有效纠正了弱势类别，我们提出的方法还为Llama-2-70B带来了显著的性能提升，尤其是在生物医学领域任务上，证明了其在小型和大型模型变体中的有效性。
URL
https://arxiv.org/abs/2503.05157
PDF
-
Label Distribution Learning-Enhanced Dual-KNN for Text Classification
2025-03-06 15:15:26Bo Yuan, Yulin Chen, Zhen Tan, Wang Jinyan, Huan Liu, Yin ZhangAbstract
Many text classification methods usually introduce external information (e.g., label descriptions and knowledge bases) to improve the classification performance. Compared to external information, some internal information generated by the model itself during training, like text embeddings and predicted label probability distributions, are exploited poorly when predicting the outcomes of some texts. In this paper, we focus on leveraging this internal information, proposing a dual $k$ nearest neighbor (D$k$NN) framework with two $k$NN modules, to retrieve several neighbors from the training set and augment the distribution of labels. For the $k$NN module, it is easily confused and may cause incorrect predictions when retrieving some nearest neighbors from noisy datasets (datasets with labeling errors) or similar datasets (datasets with similar labels). To address this issue, we also introduce a label distribution learning module that can learn label similarity, and generate a better label distribution to help models distinguish texts more effectively. This module eases model overfitting and improves final classification performance, hence enhancing the quality of the retrieved neighbors by $k$NN modules during inference. Extensive experiments on the benchmark datasets verify the effectiveness of our method.
Abstract (translated)
许多文本分类方法通常会引入外部信息（例如标签描述和知识库）来提高分类性能。相比之下，模型在训练过程中生成的内部信息（如文本嵌入和预测的概率分布），在预测某些文本的结果时被利用得较少。在这篇论文中，我们专注于利用这种内部信息，提出了一个双重$k$最近邻（D$k$NN）框架，该框架包含两个$k$近邻模块，用于从训练集中检索多个邻居并增强标签的分布。 对于$k$最近邻模块，在从有噪声的数据集（带有标注错误的数据集）或相似数据集（具有类似标签的数据集）中检索一些最近邻时，它容易混淆并且可能导致不正确的预测。为了解决这个问题，我们还引入了一个标签分布学习模块，该模块可以学习标签之间的相似性，并生成一个更好的标签分布以帮助模型更有效地区分文本。这个模块有助于缓解模型过拟合问题并提高最终的分类性能，从而在推理过程中通过$k$近邻模块增强邻居检索的质量。 我们在基准数据集上进行了广泛的实验验证了我们方法的有效性。
URL
https://arxiv.org/abs/2503.04869
PDF
