Text_Classification
-
Misspellings in Natural Language Processing: A survey
2025-01-28 10:26:04Gianluca Sperduti, Alejandro MoreoAbstract
This survey provides an overview of the challenges of misspellings in natural language processing (NLP). While often unintentional, misspellings have become ubiquitous in digital communication, especially with the proliferation of Web 2.0, user-generated content, and informal text mediums such as social media, blogs, and forums. Even if humans can generally interpret misspelled text, NLP models frequently struggle to handle it: this causes a decline in performance in common tasks like text classification and machine translation. In this paper, we reconstruct a history of misspellings as a scientific problem. We then discuss the latest advancements to address the challenge of misspellings in NLP. Main strategies to mitigate the effect of misspellings include data augmentation, double step, character-order agnostic, and tuple-based methods, among others. This survey also examines dedicated data challenges and competitions to spur progress in the field. Critical safety and ethical concerns are also examined, for example, the voluntary use of misspellings to inject malicious messages and hate speech on social networks. Furthermore, the survey explores psycholinguistic perspectives on how humans process misspellings, potentially informing innovative computational techniques for text normalization and representation. Finally, the misspelling-related challenges and opportunities associated with modern large language models are also analyzed, including benchmarks, datasets, and performances of the most prominent language models against misspellings. This survey aims to be an exhaustive resource for researchers seeking to mitigate the impact of misspellings in the rapidly evolving landscape of NLP.
Abstract (translated)
这项调查概述了自然语言处理（NLP）中错拼词的挑战。尽管通常无意为之，但错拼词在数字通信中已经变得无处不在，尤其是在Web 2.0、用户生成内容以及社交媒体、博客和论坛等非正式文本媒介兴起的情况下更是如此。虽然人类一般可以理解错误拼写的文本，但NLP模型却经常难以处理：这导致了诸如文本分类和机器翻译等常见任务性能的下降。在本文中，我们重建了错拼词作为科学问题的历史，并讨论了最新进展以应对NLP中的错拼挑战。 主要缓解错拼影响的战略包括数据增强、双步法、字符顺序无关性和元组基于的方法等等。此调查还考察了专门的数据挑战和竞赛，旨在推动该领域的发展。同时探讨了一些关键的安全与伦理关注点，例如在社交网络上故意使用错拼词来注入恶意信息和仇恨言论的问题。 此外，这项调查还探索了从心理语言学角度如何处理错拼词的人类方法，这可能有助于创新的计算技术进行文本规范化和表示。最后，分析了现代大型语言模型相关的错拼挑战与机遇，包括基准测试、数据集以及最突出的语言模型在应对错拼词方面的表现。 本调查旨在为研究者提供一份详尽资源，在不断发展的NLP领域中减轻错拼的影响。
URL
https://arxiv.org/abs/2501.16836
PDF
-
StaICC: Standardized Evaluation for Classification Task in In-context Learning
2025-01-27 00:05:12Hakaze Cho, Naoya InoueAbstract
Classification tasks are widely investigated in the In-Context Learning (ICL) paradigm. However, current efforts are evaluated on disjoint benchmarks and settings, while their performances are significantly influenced by some trivial variables, such as prompt templates, data sampling, instructions, etc., which leads to significant inconsistencies in the results reported across various literature, preventing fair comparison or meta-analysis across different papers. Therefore, this paper proposes a standardized and easy-to-use evaluation toolkit (StaICC) for in-context classification. Including, for the normal classification task, we provide StaICC-Normal, selecting 10 widely used datasets, and generating prompts with a fixed form, to mitigate the variance among the experiment implementations. To enrich the usage of our benchmark, we also provide a sub-benchmark StaICC-Diag for diagnosing ICL from several aspects, aiming for a more robust inference processing.
Abstract (translated)
在上下文学习（ICL）范式中，分类任务得到了广泛的研究。然而，当前的努力都是在相互独立的基准和设置上进行评估的，并且它们的表现受到了一些琐碎变量的影响，比如提示模板、数据采样、指令等。这导致了不同文献报告的结果存在显著不一致性，阻碍了跨论文之间的公平比较或元分析。因此，本文提出了一种标准化且易于使用的评估工具包（StaICC），用于上下文分类的评价。具体来说，对于普通的分类任务，我们提供了StaICC-Normal，选择了10个常用的基准数据集，并使用固定形式生成提示，以减少实验实现之间的差异。为了丰富我们的基准测试套件的应用范围，我们也提供了一个子基准StaICC-Diag，用于从多个角度诊断ICL，旨在促进更稳健的推理处理。
URL
https://arxiv.org/abs/2501.15708
PDF
-
Quantum-Enhanced Attention Mechanism in NLP: A Hybrid Classical-Quantum Approach
2025-01-26 18:29:06S. M. Yousuf Iqbal Tomal, Abdullah Al Shafin, Debojit Bhattacharjee, MD. Khairul Amin, Rafiad Sadat ShahirAbstract
Transformer-based models have achieved remarkable results in natural language processing (NLP) tasks such as text classification and machine translation. However, their computational complexity and resource demands pose challenges for scalability and accessibility. This research proposes a hybrid quantum-classical transformer model that integrates a quantum-enhanced attention mechanism to address these limitations. By leveraging quantum kernel similarity and variational quantum circuits (VQC), the model captures intricate token dependencies while improving computational efficiency. Experimental results on the IMDb dataset demonstrate that the quantum-enhanced model outperforms the classical baseline across all key metrics, achieving a 1.5% improvement in accuracy (65.5% vs. 64%), precision, recall, and F1 score. Statistical significance tests validate these improvements, highlighting the robustness of the quantum approach. These findings illustrate the transformative potential of quantum-enhanced attention mechanisms in optimizing NLP architectures for real-world applications.
Abstract (translated)
基于Transformer的模型在自然语言处理（NLP）任务中，如文本分类和机器翻译方面取得了显著成果。然而，这些模型由于计算复杂性和资源需求而面临可扩展性和易用性的挑战。本研究提出了一种混合量子-经典Transformer模型，该模型集成了增强型注意力机制，以解决这些问题。通过利用量子核相似度和变分量子电路（VQC），该模型能够捕捉到复杂的令牌依赖关系，并提高计算效率。在IMDb数据集上的实验结果显示，经过量子增强的模型在所有关键指标上均超越了经典基线模型，准确率提高了1.5%（从64%提升至65.5%），同时精度、召回率和F1分数也有所改善。统计显著性测试验证了这些改进，并突显出量子方法的鲁棒性。这些发现展示了量子增强注意力机制在优化NLP架构以应用于实际场景中的变革潜力。
URL
https://arxiv.org/abs/2501.15630
PDF
-
Idiom Detection in Sorani Kurdish Texts
2025-01-24 14:31:30Skala Kamaran Omer, Hossein HassaniAbstract
Idiom detection using Natural Language Processing (NLP) is the computerized process of recognizing figurative expressions within a text that convey meanings beyond the literal interpretation of the words. While idiom detection has seen significant progress across various languages, the Kurdish language faces a considerable research gap in this area despite the importance of idioms in tasks like machine translation and sentiment analysis. This study addresses idiom detection in Sorani Kurdish by approaching it as a text classification task using deep learning techniques. To tackle this, we developed a dataset containing 10,580 sentences embedding 101 Sorani Kurdish idioms across diverse contexts. Using this dataset, we developed and evaluated three deep learning models: KuBERT-based transformer sequence classification, a Recurrent Convolutional Neural Network (RCNN), and a BiLSTM model with an attention mechanism. The evaluations revealed that the transformer model, the fine-tuned BERT, consistently outperformed the others, achieving nearly 99% accuracy while the RCNN achieved 96.5% and the BiLSTM 80%. These results highlight the effectiveness of Transformer-based architectures in low-resource languages like Kurdish. This research provides a dataset, three optimized models, and insights into idiom detection, laying a foundation for advancing Kurdish NLP.
Abstract (translated)
使用自然语言处理（NLP）进行成语检测是通过计算机识别文本中表达超出字面意义的比喻性表达的过程。尽管在多种语言上已经取得了显著进展，但库尔德语（尤其是索拉尼库尔德语）在这个领域的研究仍然存在较大空白，这不利于机器翻译和情感分析等任务的发展。这项研究表明如何使用深度学习技术将索拉尼库尔德语的成语检测转化为文本分类问题。 为了实现这一目标，我们创建了一个包含10,580个句子的数据集，这些句子中嵌入了101条索拉尼库尔德语成语，并涵盖了不同的上下文。利用这个数据集，我们开发并评估了三种深度学习模型：基于KuBERT的Transformer序列分类、递归卷积神经网络（RCNN）以及带有注意力机制的双层长短期记忆网络（BiLSTM）。在实验中，变压器模型（微调后的BERT模型）的表现始终优于其他两种模型，在准确性上接近99%，而RCNN和BiLSTM分别达到了96.5% 和80%。 这些结果突显了基于Transformer架构的有效性，尤其是在像库尔德语这样的低资源语言环境中。这项研究不仅提供了一个数据集、三种优化后的模型以及有关成语检测的见解，并为推动库尔德语自然语言处理的进步奠定了基础。
URL
https://arxiv.org/abs/2501.14528
PDF
-
A Comprehensive Framework for Semantic Similarity Detection Using Transformer Architectures and Enhanced Ensemble Techniques
2025-01-24 07:07:37Lifu Gao, Qi Zhang, Ziwei LiuAbstract
Detecting AI-generated text, especially in short-context documents, is difficult because there is not enough context for accurate classification. This paper presents a new teacher-student model that uses domain adaptation and data augmentation to solve these problems. The teacher model, which combines DeBERTa-v3-large and Mamba-790m, learns semantic knowledge through domain-specific fine-tuning. The student model handles short-context text more efficiently. The system uses a Mean Squared Error (MSE) loss function to guide the student's learning, improving both accuracy and efficiency. Also, data augmentation methods like spelling correction and error injection make the model more robust. Experimental results show that this approach works better than baseline methods, proving its usefulness for real-time AI-generated text detection and other text classification tasks.
Abstract (translated)
检测AI生成文本，尤其是在短文档中，由于缺乏足够的上下文信息来进行准确分类而变得困难。本文介绍了一种新的师生模型，该模型结合领域适应和数据增强技术来解决这些问题。教师模型融合了DeBERTa-v3-large和Mamba-790m，并通过特定领域的微调学习语义知识。学生模型则更高效地处理短文本上下文的检测任务。系统采用均方误差（Mean Squared Error，MSE）损失函数引导学生的训练过程，从而提高准确性和效率。此外，数据增强方法如拼写纠错和错误注入使模型更加健壮。实验结果表明，该方法比基线方法表现更好，证明了其在实时AI生成文本检测及其他文本分类任务中的实用性。
URL
https://arxiv.org/abs/2501.14288
PDF
-
A Study of the Plausibility of Attention between RNN Encoders in Natural Language Inference
2025-01-23 15:11:27Duc Hau Nguyen, Duc Hau Nguyen, Pascale S\'ebillotAbstract
Attention maps in neural models for NLP are appealing to explain the decision made by a model, hopefully emphasizing words that justify the decision. While many empirical studies hint that attention maps can provide such justification from the analysis of sound examples, only a few assess the plausibility of explanations based on attention maps, i.e., the usefulness of attention maps for humans to understand the decision. These studies furthermore focus on text classification. In this paper, we report on a preliminary assessment of attention maps in a sentence comparison task, namely natural language inference. We compare the cross-attention weights between two RNN encoders with human-based and heuristic-based annotations on the eSNLI corpus. We show that the heuristic reasonably correlates with human annotations and can thus facilitate evaluation of plausible explanations in sentence comparison tasks. Raw attention weights however remain only loosely related to a plausible explanation.
Abstract (translated)
神经模型中的注意力图（Attention maps）在解释模型决策方面具有吸引力，理想情况下应强调那些能为决策提供依据的词语。虽然许多实证研究表明，通过对典型示例的分析可以得出这样的结论，即注意力图能够为模型决策提供合理说明，但只有少数研究评估了基于注意力图的解释的实际有效性，也就是这些解释对于人类理解模型决策是否有用。而且这些研究主要集中在文本分类任务上。 本文报告了一项初步研究，在句子比较任务（特别是自然语言推理）中对注意力图进行了评估。我们对比了两个RNN编码器之间的交叉注意权重，并将其与eSNLI语料库中的基于人的注释和基于启发式的注释进行对照。结果表明，基于启发式的方法合理地与人类的标注相关联，因此可以在句子比较任务中促进可接受解释的评估。然而，原始注意力权重仍然仅与合理的解释存在松散的相关性。
URL
https://arxiv.org/abs/2501.13735
PDF
-
Multi-Level Attention and Contrastive Learning for Enhanced Text Classification with an Optimized Transformer
2025-01-23 08:32:27Jia Gao, Guiran Liu, Binrong Zhu, Shicheng Zhou, Hongye Zheng, Xiaoxuan LiaoAbstract
This paper studies a text classification algorithm based on an improved Transformer to improve the performance and efficiency of the model in text classification tasks. Aiming at the shortcomings of the traditional Transformer model in capturing deep semantic relationships and optimizing computational complexity, this paper introduces a multi-level attention mechanism and a contrastive learning strategy. The multi-level attention mechanism effectively models the global semantics and local features in the text by combining global attention with local attention; the contrastive learning strategy enhances the model's ability to distinguish between different categories by constructing positive and negative sample pairs while improving the classification effect. In addition, in order to improve the training and inference efficiency of the model on large-scale text data, this paper designs a lightweight module to optimize the feature transformation process and reduce the computational cost. Experimental results on the dataset show that the improved Transformer model outperforms the comparative models such as BiLSTM, CNN, standard Transformer, and BERT in terms of classification accuracy, F1 score, and recall rate, showing stronger semantic representation ability and generalization performance. The method proposed in this paper provides a new idea for algorithm optimization in the field of text classification and has good application potential and practical value. Future work will focus on studying the performance of this model in multi-category imbalanced datasets and cross-domain tasks and explore the integration wi
Abstract (translated)
这篇论文研究了一种基于改进Transformer的文本分类算法，以提高模型在文本分类任务中的性能和效率。针对传统Transformer模型在捕捉深层次语义关系及优化计算复杂度方面的不足，本文引入了多层次注意力机制以及对比学习策略。多层次注意力机制通过结合全局注意力与局部注意力有效建模文本的全局语义和局部特征；对比学习策略则通过构造正负样本对来增强模型区分不同类别的能力，并提升分类效果的同时改进模型性能。 此外，为了提高大规模文本数据上模型训练及推理效率，本文设计了一个轻量级模块以优化特征转换过程并降低计算成本。实验结果表明，在测试集上的分类准确率、F1值和召回率等方面，改进后的Transformer模型优于BiLSTM、CNN、标准Transformer以及BERT等对比模型，显示出更强的语义表示能力和泛化性能。 该文中提出的这种方法为文本分类领域的算法优化提供了新的思路，并具有良好的应用潜力与实际价值。未来的工作将集中于研究此模型在多类别不平衡数据集及跨域任务中的表现，并探索与其他技术的集成应用可能性。
URL
https://arxiv.org/abs/2501.13467
PDF
-
Regularization, Semi-supervision, and Supervision for a Plausible Attention-Based Explanation
2025-01-22 10:17:20Duc Hau Nguyen, Cyrielle Mallart, Guillaume Gravier, Pascale S\'ebillotAbstract
Attention mechanism is contributing to the majority of recent advances in machine learning for natural language processing. Additionally, it results in an attention map that shows the proportional influence of each input in its decision. Empirical studies postulate that attention maps can be provided as an explanation for model output. However, it is still questionable to ask whether this explanation helps regular people to understand and accept the model output (the plausibility of the explanation). Recent studies show that attention weights in the RNN encoders are hardly plausible because they spread on input tokens. We thus propose 3 additional constraints to the learning objective function to improve the plausibility of the attention map: regularization to increase the attention weight sparsity, semi-supervision to supervise the map by a heuristic and supervision by human annotation. Results show that all techniques can improve the attention map plausibility at some level. We also observe that specific instructions for human annotation might have a negative effect on classification performance. Beyond the attention map, the result of experiments on text classification tasks also shows that no matter how the constraint brings the gain, the contextualization layer plays a crucial role in finding the right space for finding plausible tokens.
Abstract (translated)
注意力机制在自然语言处理领域的机器学习最近进展中发挥了重要作用。此外，它生成了一个显示每个输入在其决策中的相对影响力的关注图（attention map）。实证研究表明，可以将这些关注图作为模型输出的解释提供给人们。然而，仍值得探讨的是，这种解释是否有助于普通人理解和接受模型的输出结果（即解释的有效性）。 最近的研究表明，在循环神经网络（RNN）编码器中的注意力权重难以具有有效性，因为它们分散在输入令牌上。因此，我们提出了三种附加约束来改善关注图的有效性：增加注意权重稀疏性的正则化、通过启发式方法监督地图的半监督学习以及由人工注释进行监督。 实验结果显示，所有这些技术都在一定程度上提高了注意力图的有效性。同时观察到，特定的人类标注指导可能对分类性能产生负面影响。除了关注图之外，在文本分类任务上的实验证明，无论哪种约束带来了多少收益，上下文化层（contextualization layer）在找到合适的令牌空间以提高有效性方面起着至关重要的作用。
URL
https://arxiv.org/abs/2501.12775
PDF
-
Multimodal Marvels of Deep Learning in Medical Diagnosis: A Comprehensive Review of COVID-19 Detection
2025-01-16 12:38:49Md Shofiqul Islama, Khondokar Fida Hasanc, Hasibul Hossain Shajeebd, Humayan Kabir Ranae, Md Saifur Rahmand, Md Munirul Hasanb, AKM Azadf, Ibrahim Abdullahg, Mohammad Ali MoniAbstract
This study presents a comprehensive review of the potential of multimodal deep learning (DL) in medical diagnosis, using COVID-19 as a case example. Motivated by the success of artificial intelligence applications during the COVID-19 pandemic, this research aims to uncover the capabilities of DL in disease screening, prediction, and classification, and to derive insights that enhance the resilience, sustainability, and inclusiveness of science, technology, and innovation systems. Adopting a systematic approach, we investigate the fundamental methodologies, data sources, preprocessing steps, and challenges encountered in various studies and implementations. We explore the architecture of deep learning models, emphasising their data-specific structures and underlying algorithms. Subsequently, we compare different deep learning strategies utilised in COVID-19 analysis, evaluating them based on methodology, data, performance, and prerequisites for future research. By examining diverse data types and diagnostic modalities, this research contributes to scientific understanding and knowledge of the multimodal application of DL and its effectiveness in diagnosis. We have implemented and analysed 11 deep learning models using COVID-19 image, text, and speech (ie, cough) data. Our analysis revealed that the MobileNet model achieved the highest accuracy of 99.97% for COVID-19 image data and 93.73% for speech data (i.e., cough). However, the BiGRU model demonstrated superior performance in COVID-19 text classification with an accuracy of 99.89%. The broader implications of this research suggest potential benefits for other domains and disciplines that could leverage deep learning techniques for image, text, and speech analysis.
Abstract (translated)
这项研究提供了多模态深度学习（DL）在医学诊断中潜在应用的全面回顾，以COVID-19为例。鉴于人工智能技术在新冠疫情期间的成功应用，本研究旨在揭示深度学习在疾病筛查、预测和分类方面的潜力，并从中获得有助于增强科学、技术和创新体系韧性、可持续性和包容性的见解。采用系统方法，我们探讨了各种研究与实施中遇到的基本方法论、数据来源、预处理步骤以及所面临的挑战。我们还探索了深度学习模型的架构，强调其特定于数据的结构及其基础算法。接下来，我们将比较在COVID-19分析中使用的不同深度学习策略，并根据方法学、数据、性能和未来研究的需求对其进行评估。 通过考察不同类型的数据及诊断模式，本研究为多模态应用下的DL科学理解和知识贡献了力量，并探讨其在诊断中的有效性。我们实施并分析了11种基于COVID-19图像、文本以及语音（即咳嗽）数据的深度学习模型。我们的分析表明，MobileNet模型对COVID-19图像数据实现了最高精度为99.97%，而针对语音数据（如咳嗽）的准确率达到了93.73%。然而，在COVID-19文本分类中，BiGRU模型表现出色，其准确性达到99.89%。 这项研究更广泛的含义在于，它可能对其他领域和学科产生潜在益处，这些领域和学科可以利用深度学习技术进行图像、文本以及语音分析。
URL
https://arxiv.org/abs/2501.09506
PDF
-
A Simple Graph Contrastive Learning Framework for Short Text Classification
2025-01-16 00:35:56Yonghao Liu, Fausto Giunchiglia, Lan Huang, Ximing Li, Xiaoyue Feng, Renchu GuanAbstract
Short text classification has gained significant attention in the information age due to its prevalence and real-world applications. Recent advancements in graph learning combined with contrastive learning have shown promising results in addressing the challenges of semantic sparsity and limited labeled data in short text classification. However, existing models have certain limitations. They rely on explicit data augmentation techniques to generate contrastive views, resulting in semantic corruption and noise. Additionally, these models only focus on learning the intrinsic consistency between the generated views, neglecting valuable discriminative information from other potential views. To address these issues, we propose a Simple graph contrastive learning framework for Short Text Classification (SimSTC). Our approach involves performing graph learning on multiple text-related component graphs to obtain multi-view text embeddings. Subsequently, we directly apply contrastive learning on these embeddings. Notably, our method eliminates the need for data augmentation operations to generate contrastive views while still leveraging the benefits of multi-view contrastive learning. Despite its simplicity, our model achieves outstanding performance, surpassing large language models on various datasets.
Abstract (translated)
在信息时代，由于其普遍性和实际应用价值，短文本分类已获得了广泛关注。近期，图学习与对比学习相结合的技术，在解决短文本分类中的语义稀疏性和标注数据不足的挑战方面显示出巨大潜力。然而，现有的模型存在一定的局限性：它们依赖于显式的数据增强技术来生成对比视图，这会导致语义污染和噪声；此外，这些模型仅关注于从生成的视图中学习内在一致性，而忽视了其他潜在视图中的有价值的区别信息。 为了解决这些问题，我们提出了一种用于短文本分类的简单图对比学习框架（SimSTC）。我们的方法包括在多个与文本相关的组件图上执行图学习以获取多视角文本嵌入，随后直接在这些建模后的嵌入上应用对比学习。特别值得注意的是，我们的方法消除了生成对比视图所需的数据增强操作，同时仍然利用了多视图对比学习的好处。尽管该模型结构简单，但在各种数据集上的性能表现却非常出色，并且超过了大型语言模型的水平。
URL
https://arxiv.org/abs/2501.09219
PDF
-
Boosting Short Text Classification with Multi-Source Information Exploration and Dual-Level Contrastive Learning
2025-01-16 00:26:15Yonghao Liu, Mengyu Li, Wei Pang, Fausto Giunchiglia, Lan Huang, Xiaoyue Feng, Renchu GuanAbstract
Short text classification, as a research subtopic in natural language processing, is more challenging due to its semantic sparsity and insufficient labeled samples in practical scenarios. We propose a novel model named MI-DELIGHT for short text classification in this work. Specifically, it first performs multi-source information (i.e., statistical information, linguistic information, and factual information) exploration to alleviate the sparsity issues. Then, the graph learning approach is adopted to learn the representation of short texts, which are presented in graph forms. Moreover, we introduce a dual-level (i.e., instance-level and cluster-level) contrastive learning auxiliary task to effectively capture different-grained contrastive information within massive unlabeled data. Meanwhile, previous models merely perform the main task and auxiliary tasks in parallel, without considering the relationship among tasks. Therefore, we introduce a hierarchical architecture to explicitly model the correlations between tasks. We conduct extensive experiments across various benchmark datasets, demonstrating that MI-DELIGHT significantly surpasses previous competitive models. It even outperforms popular large language models on several datasets.
Abstract (translated)
短文本分类作为自然语言处理领域的研究子课题，因其语义稀疏性和实际场景中标签样本不足而更具挑战性。在本工作中，我们提出了一种名为MI-DELIGHT的新模型，用于解决短文本分类问题。具体来说，该模型首先进行多源信息（即统计信息、语言学信息和事实信息）的探索以缓解语义稀疏性的问题。然后，采用图学习方法来学习用图形式表示的短文本的表征。此外，我们引入了一种双层次（实例级和聚类级）对比学习辅助任务，有效捕获大量未标记数据中的不同粒度的对比信息。同时，先前的模型仅在并行执行主要任务与辅助任务时，并不考虑各任务之间的关系。因此，我们引入了分层架构以明确建模各个任务间的相关性。我们在多个基准数据集上进行了广泛的实验，结果表明MI-DELIGHT显著超越了以前的竞争模型，在某些数据集中甚至超过了流行的大规模语言模型的性能。
URL
https://arxiv.org/abs/2501.09214
PDF
-
Augmenting Human-Annotated Training Data with Large Language Model Generation and Distillation in Open-Response Assessment
2025-01-15 20:13:46Conrad Borchers, Danielle R. Thomas, Jionghao Lin, Ralph Abboud, Kenneth R. KoedingerAbstract
Large Language Models (LLMs) like GPT-4o can help automate text classification tasks at low cost and scale. However, there are major concerns about the validity and reliability of LLM outputs. By contrast, human coding is generally more reliable but expensive to procure at scale. In this study, we propose a hybrid solution to leverage the strengths of both. We combine human-coded data and synthetic LLM-produced data to fine-tune a classical machine learning classifier, distilling both into a smaller BERT model. We evaluate our method on a human-coded test set as a validity measure for LLM output quality. In three experiments, we systematically vary LLM-generated samples' size, variety, and consistency, informed by best practices in LLM tuning. Our findings indicate that augmenting datasets with synthetic samples improves classifier performance, with optimal results achieved at an 80% synthetic to 20% human-coded data ratio. Lower temperature settings of 0.3, corresponding to less variability in LLM generations, produced more stable improvements but also limited model learning from augmented samples. In contrast, higher temperature settings (0.7 and above) introduced greater variability in performance estimates and, at times, lower performance. Hence, LLMs may produce more uniform output that classifiers overfit to earlier or produce more diverse output that runs the risk of deteriorating model performance through information irrelevant to the prediction task. Filtering out inconsistent synthetic samples did not enhance performance. We conclude that integrating human and LLM-generated data to improve text classification models in assessment offers a scalable solution that leverages both the accuracy of human coding and the variety of LLM outputs.
Abstract (translated)
大型语言模型（LLMs）如GPT-4可以以低成本和大规模的方式自动化文本分类任务，但其输出的有效性和可靠性存在重大问题。相比之下，人工编码的数据通常更可靠，但在大规模获取时成本高昂。在本研究中，我们提出了一种混合解决方案，旨在结合两者的优势。我们将人工编码数据与LLM生成的合成数据结合起来，对经典的机器学习分类器进行微调，并将这些数据浓缩到一个较小的BERT模型中。我们在一个人工编码的测试集上评估了我们的方法的有效性，作为衡量LLM输出质量的标准。在三项实验中，我们系统地改变了LLM生成样本的数量、种类和一致性，参考了最佳的LLM调优实践。我们的发现表明，通过使用合成样本增强数据集可以提高分类器的性能，在80%合成样本与20%人工编码数据的比例下达到最优结果。 较低的温度设置（如0.3）对应于更少的生成变异性，并产生了更加稳定的改进效果，但也限制了模型从扩充的数据中学习。相比之下，较高的温度设置（0.7及以上）引入了更大的性能估计变动性，在某些情况下甚至导致性能下降。因此，LLM可能产生分类器会过度拟合的较为统一的输出，或者生成多样化的输出从而带来风险，即通过与预测任务无关的信息降低模型性能。 过滤不一致的合成样本并未提高性能表现。综上所述，结合人工和LLM生成的数据以改进评估中的文本分类模型提供了一种可扩展的解决方案，该方案利用了人工编码的准确性以及LLM输出的多样性。
URL
https://arxiv.org/abs/2501.09126
PDF
-
Large Language Models For Text Classification: Case Study And Comprehensive Review
2025-01-14 22:02:38Arina Kostina, Marios D. Dikaiakos, Dimosthenis Stefanidis, George PallisAbstract
Unlocking the potential of Large Language Models (LLMs) in data classification represents a promising frontier in natural language processing. In this work, we evaluate the performance of different LLMs in comparison with state-of-the-art deep-learning and machine-learning models, in two different classification scenarios: i) the classification of employees' working locations based on job reviews posted online (multiclass classification), and 2) the classification of news articles as fake or not (binary classification). Our analysis encompasses a diverse range of language models differentiating in size, quantization, and architecture. We explore the impact of alternative prompting techniques and evaluate the models based on the weighted F1-score. Also, we examine the trade-off between performance (F1-score) and time (inference response time) for each language model to provide a more nuanced understanding of each model's practical applicability. Our work reveals significant variations in model responses based on the prompting strategies. We find that LLMs, particularly Llama3 and GPT-4, can outperform traditional methods in complex classification tasks, such as multiclass classification, though at the cost of longer inference times. In contrast, simpler ML models offer better performance-to-time trade-offs in simpler binary classification tasks.
Abstract (translated)
解锁大型语言模型（LLMs）在数据分类中的潜力代表了自然语言处理领域的一个有前景的前沿方向。在这项工作中，我们评估了不同LLM与最先进的深度学习和机器学习模型相比，在两种不同的分类场景下的性能：i) 根据在线发布的职位评价来分类员工的工作地点（多类分类），以及 2) 将新闻文章分类为假新闻或非假新闻（二元分类）。我们的分析涵盖了各种在大小、量化和架构上有所区别的语言模型。我们探讨了不同的提示技术的影响，并根据加权F1分数对这些模型进行评估。此外，我们还考察了性能（F1得分）与时间（推理响应时间）之间的权衡关系，以提供每种语言模型实际应用中更细致的理解。我们的研究揭示了基于不同提示策略的模型响应存在显著差异。我们发现，特别是Llama3和GPT-4这样的LLM在复杂的分类任务（如多类分类）上可以超越传统方法，尽管这会带来推理时间较长的问题。相反，在较为简单的二元分类任务中，简单的机器学习模型提供了更好的性能与时间的权衡关系。
URL
https://arxiv.org/abs/2501.08457
PDF
-
READ: Reinforcement-based Adversarial Learning for Text Classification with Limited Labeled Data
2025-01-14 11:39:55Rohit Sharma, Shanu Kumar, Avinash KumarAbstract
Pre-trained transformer models such as BERT have shown massive gains across many text classification tasks. However, these models usually need enormous labeled data to achieve impressive performances. Obtaining labeled data is often expensive and time-consuming, whereas collecting unlabeled data using some heuristics is relatively much cheaper for any task. Therefore, this paper proposes a method that encapsulates reinforcement learning-based text generation and semi-supervised adversarial learning approaches in a novel way to improve the model's performance. Our method READ, Reinforcement-based Adversarial learning, utilizes an unlabeled dataset to generate diverse synthetic text through reinforcement learning, improving the model's generalization capability using adversarial learning. Our experimental results show that READ outperforms the existing state-of-art methods on multiple datasets.
Abstract (translated)
预训练的Transformer模型（如BERT）在许多文本分类任务中展现了巨大的优势。然而，这些模型通常需要大量的标注数据才能达到令人印象深刻的性能水平。获取标注数据往往既昂贵又耗时，而使用一些启发式方法收集未标记的数据对于任何任务来说都相对便宜得多。因此，本文提出了一种创新的方法，该方法将基于强化学习的文本生成和半监督对抗性学习相结合，以提高模型的表现。 我们的方法称为READ（Reinforcement-based Adversarial Learning），它利用未标注的数据集通过强化学习来生成多样化的合成文本，并使用对抗性学习提升模型的泛化能力。实验结果表明，READ在多个数据集上优于现有的最先进方法。
URL
https://arxiv.org/abs/2501.08035
PDF
-
Universal Training of Neural Networks to Achieve Bayes Optimal Classification Accuracy
2025-01-13 23:55:11Mohammadreza Tavasoli Naeini, Ali Bereyhi, Morteza Noshad, Ben Liang, Alfred O. Hero IIIAbstract
This work invokes the notion of $f$-divergence to introduce a novel upper bound on the Bayes error rate of a general classification task. We show that the proposed bound can be computed by sampling from the output of a parameterized model. Using this practical interpretation, we introduce the Bayes optimal learning threshold (BOLT) loss whose minimization enforces a classification model to achieve the Bayes error rate. We validate the proposed loss for image and text classification tasks, considering MNIST, Fashion-MNIST, CIFAR-10, and IMDb datasets. Numerical experiments demonstrate that models trained with BOLT achieve performance on par with or exceeding that of cross-entropy, particularly on challenging datasets. This highlights the potential of BOLT in improving generalization.
Abstract (translated)
这项工作引入了$f$-散度的概念，提出了一种对一般分类任务的贝叶斯错误率的新上界估计。我们展示了所提出的界限可以通过从参数化模型的输出中采样来计算得出。基于这一实用解释，我们提出了贝叶斯最优学习阈值（BOLT）损失函数，其最小化能够促使分类模型达到贝叶斯错误率。我们在图像和文本分类任务中验证了该损失函数的有效性，考虑的数据集包括MNIST、Fashion-MNIST、CIFAR-10以及IMDb数据集。数值实验表明，在训练时使用BOLT的模型在性能上可以与交叉熵相媲美或超越交叉熵，尤其是在挑战性的数据集上更为明显。这突显了BOLT在提升泛化能力方面的潜力。
URL
https://arxiv.org/abs/2501.07754
PDF
-
Synthetic Feature Augmentation Improves Generalization Performance of Language Models
2025-01-11 04:31:18Ashok Choudhary, Cornelius Thiels, Hojjat SalehinejadAbstract
Training and fine-tuning deep learning models, especially large language models (LLMs), on limited and imbalanced datasets poses substantial challenges. These issues often result in poor generalization, where models overfit to dominant classes and underperform on minority classes, leading to biased predictions and reduced robustness in real-world applications. To overcome these challenges, we propose augmenting features in the embedding space by generating synthetic samples using a range of techniques. By upsampling underrepresented classes, this method improves model performance and alleviates data imbalance. We validate the effectiveness of this approach across multiple open-source text classification benchmarks, demonstrating its potential to enhance model robustness and generalization in imbalanced data scenarios.
Abstract (translated)
在有限且不平衡的数据集上训练和微调深度学习模型，尤其是大型语言模型（LLMs），面临着重大挑战。这些问题常常导致模型泛化能力差：过度拟合到主要类别并在次要类别中表现不佳，从而导致预测偏见并降低实际应用中的鲁棒性。为克服这些挑战，我们提出通过在嵌入空间生成合成样本的方法来增强特征，以此来提升未充分代表类别的数据量。这种方法可以提高模型性能，并缓解数据不平衡的问题。我们在多个开源文本分类基准测试中验证了该方法的有效性，展示了其在不均衡数据场景下增强模型鲁棒性和泛化能力的潜力。
URL
https://arxiv.org/abs/2501.06434
PDF
-
The Impact of Model Scaling on Seen and Unseen Language Performance
2025-01-10 00:10:21Rhitabrat Pokharel, Sina Bagheri Nezhad, Ameeta Agrawal, Suresh SinghAbstract
The rapid advancement of Large Language Models (LLMs), particularly those trained on multilingual corpora, has intensified the need for a deeper understanding of their performance across a diverse range of languages and model sizes. Our research addresses this critical need by studying the performance and scaling behavior of multilingual LLMs in text classification and machine translation tasks across 204 languages. We systematically examine both seen and unseen languages across three model families of varying sizes in zero-shot and few-shot settings. Our findings show significant differences in scaling behavior between zero-shot and two-shot scenarios, with striking disparities in performance between seen and unseen languages. Model scale has little effect on zero-shot performance, which remains mostly flat. However, in two-shot settings, larger models show clear linear improvements in multilingual text classification. For translation tasks, however, only the instruction-tuned model showed clear benefits from scaling. Our analysis also suggests that overall resource levels, not just the proportions of pretraining languages, are better predictors of model performance, shedding light on what drives multilingual LLM effectiveness.
Abstract (translated)
大型语言模型（LLMs）的迅速发展，尤其是那些基于多语种语料库训练的模型，增强了对其在不同语言和规模范围内性能进行深入理解的需求。我们的研究通过探讨204种语言中多语种LLMs在文本分类和机器翻译任务中的表现及扩展行为，来解决这一关键需求。我们系统地考察了三种大小不同的模型家族在零样本和少量样本设置下已见与未见过的语言的性能。 我们的发现表明，在零样本和两样本情景之间存在显著的扩展行为差异，并且已见语言与未见过语言之间的表现差距明显。对于零样本情境，模型规模对性能影响不大，大多数情况下保持不变；然而在两样本设定中，较大的模型在多语种文本分类方面表现出明显的线性改进。但对于翻译任务而言，只有经过指令调优的模型从扩展中明确受益。 我们的分析还表明，整体资源水平（而不仅仅是预训练语言的比例）是更好的性能预测指标，这为理解驱动多语言LLMs有效性的因素提供了洞见。
URL
https://arxiv.org/abs/2501.05629
PDF
-
AgoraSpeech: A multi-annotated comprehensive dataset of political discourse through the lens of humans and AI
2025-01-09 18:17:59Pavlos Sermpezis, Stelios Karamanidis, Eva Paraschou, Ilias Dimitriadis, Sofia Yfantidou, Filitsa-Ioanna Kouskouveli, Thanasis Troboukis, Kelly Kiki, Antonis Galanopoulos, Athena VakaliAbstract
Political discourse datasets are important for gaining political insights, analyzing communication strategies or social science phenomena. Although numerous political discourse corpora exist, comprehensive, high-quality, annotated datasets are scarce. This is largely due to the substantial manual effort, multidisciplinarity, and expertise required for the nuanced annotation of rhetorical strategies and ideological contexts. In this paper, we present AgoraSpeech, a meticulously curated, high-quality dataset of 171 political speeches from six parties during the Greek national elections in 2023. The dataset includes annotations (per paragraph) for six natural language processing (NLP) tasks: text classification, topic identification, sentiment analysis, named entity recognition, polarization and populism detection. A two-step annotation was employed, starting with ChatGPT-generated annotations and followed by exhaustive human-in-the-loop validation. The dataset was initially used in a case study to provide insights during the pre-election period. However, it has general applicability by serving as a rich source of information for political and social scientists, journalists, or data scientists, while it can be used for benchmarking and fine-tuning NLP and large language models (LLMs).
Abstract (translated)
政治话语数据集对于获取政治见解、分析沟通策略或社会科学研究现象至关重要。尽管存在大量政治话语语料库，但全面且高质量的标注数据集却十分稀缺。这主要是因为对修辞策略和意识形态背景进行细致标注需要巨大的人工努力、多学科知识以及专业知识。 在本文中，我们介绍了AgoraSpeech，这是一个精心整理的数据集，包含2023年希腊全国选举期间六党派的171份政治演讲文本。该数据集包含了针对六个自然语言处理（NLP）任务的标注（每段落为单位）：文本分类、主题识别、情感分析、命名实体识别、极化和民粹主义检测。 为了构建这个数据集，我们采用了两步注释方法：首先利用ChatGPT生成初步注释，然后进行详尽的人工验证。最初，该数据集被用于案例研究，在选举前提供见解。然而，它具有广泛的适用性，可作为政治和社会科学家、记者或数据科学家的信息丰富来源，并可用于基准测试和微调NLP和大型语言模型（LLMs）。
URL
https://arxiv.org/abs/2501.06265
PDF
-
Spatial Information Integration in Small Language Models for Document Layout Generation and Classification
2025-01-09 17:20:00Pablo Melendez, Clemens HavasAbstract
Document layout understanding is a field of study that analyzes the spatial arrangement of information in a document hoping to understand its structure and layout. Models such as LayoutLM (and its subsequent iterations) can understand semi-structured documents with SotA results; however, the lack of open semi-structured data is a limitation in itself. While semi-structured data is common in everyday life (balance sheets, purchase orders, receipts), there is a lack of public datasets for training machine learning models for this type of document. In this investigation we propose a method to generate new, synthetic, layout information that can help overcoming this data shortage. According to our results, the proposed method performs better than LayoutTransformer, another popular layout generation method. We also show that, in some scenarios, text classification can improve when supported by bounding box information.
Abstract (translated)
文档布局理解是一个研究领域，它分析文档中信息的空间排列，旨在了解其结构和布局。像LayoutLM及其后续版本这样的模型可以以最先进的结果来理解和处理半结构化文档；然而，缺乏公开的半结构化数据本身就是一个限制。尽管在日常生活中，半结构化的数据非常普遍（如资产负债表、采购订单、收据），但用于训练此类文档机器学习模型的公共数据集却很匮乏。在这项研究中，我们提出了一种生成新的合成布局信息的方法，以帮助克服这一数据短缺问题。根据我们的结果，所提出的方法优于另一种流行的布局生成方法LayoutTransformer。此外，我们还表明，在某些场景下，文本分类在有边界框信息支持的情况下可以得到改进。
URL
https://arxiv.org/abs/2501.05497
PDF
-
The textit{Questio de aqua et terra}: A Computational Authorship Verification Study
2025-01-07 18:42:05Martina Leocata, Alejandro Moreo, Fabrizio SebastianiAbstract
The Questio de aqua et terra is a cosmological treatise traditionally attributed to Dante Alighieri. However, the authenticity of this text is controversial, due to discrepancies with Dante's established works and to the absence of contemporary references. This study investigates the authenticity of the Questio via computational authorship verification (AV), a class of techniques which combine supervised machine learning and stylometry. We build a family of AV systems and assemble a corpus of 330 13th- and 14th-century Latin texts, which we use to comparatively evaluate the AV systems through leave-one-out cross-validation. Our best-performing system achieves high verification accuracy (F1=0.970) despite the heterogeneity of the corpus in terms of textual genre. The key contribution to the accuracy of this system is shown to come from Distributional Random Oversampling (DRO), a technique specially tailored to text classification which is here used for the first time in AV. The application of the AV system to the Questio returns a highly confident prediction concerning its authenticity. These findings contribute to the debate on the authorship of the Questio, and highlight DRO's potential in the application of AV to cultural heritage.
Abstract (translated)
《水与地的问题》是一篇传统上归于但丁·阿利吉耶里的宇宙学著作。然而，由于该文本与但丁已知作品存在矛盾以及缺乏同时代的引用证据，其真实性备受争议。本研究通过计算作者身份验证（AV）方法来探讨《水与地的问题》的真实性。这种方法结合了监督机器学习和文体测量技术。我们构建了一组AV系统，并汇集了一个包含330篇13世纪至14世纪拉丁文文本的语料库，利用留一交叉验证法对这些系统进行比较评估。性能最佳的系统实现了高准确率（F1=0.970），尽管语料库在文体类型方面存在多样性。该系统准确性提高的关键在于分布随机过采样技术（DRO），这是一种专门用于文本分类的方法，首次在此领域应用于AV研究中。我们将AV系统应用到《水与地的问题》上，得到一个关于其真实性的高度自信的预测结果。这些发现有助于作者身份争议的讨论，并突显了DRO在文化遗产领域的AV应用潜力。
URL
https://arxiv.org/abs/2501.05480
PDF
