Text_Classification
-
Generative AI for Research Data Processing: Lessons Learnt From Three Use Cases
2025-04-22 12:21:07Modhurita Mitra, Martine G. de Vos, Nicola Cortinovis, Dawa OmettoAbstract
There has been enormous interest in generative AI since ChatGPT was launched in 2022. However, there are concerns about the accuracy and consistency of the outputs of generative AI. We have carried out an exploratory study on the application of this new technology in research data processing. We identified tasks for which rule-based or traditional machine learning approaches were difficult to apply, and then performed these tasks using generative AI. We demonstrate the feasibility of using the generative AI model Claude 3 Opus in three research projects involving complex data processing tasks: 1) Information extraction: We extract plant species names from historical seedlists (catalogues of seeds) published by botanical gardens. 2) Natural language understanding: We extract certain data points (name of drug, name of health indication, relative effectiveness, cost-effectiveness, etc.) from documents published by Health Technology Assessment organisations in the EU. 3) Text classification: We assign industry codes to projects on the crowdfunding website Kickstarter. We share the lessons we learnt from these use cases: How to determine if generative AI is an appropriate tool for a given data processing task, and if so, how to maximise the accuracy and consistency of the results obtained.
Abstract (translated)
自2022年ChatGPT推出以来，生成式人工智能引起了极大的兴趣。然而，人们对生成式AI输出的准确性和一致性表示担忧。我们进行了一项探索性研究，探讨这项新技术在研究数据处理中的应用情况。我们确定了规则基础或传统机器学习方法难以应对的任务，并使用生成式AI执行这些任务。我们在三个涉及复杂数据处理的研究项目中展示了Claude 3 Opus这一生成式AI模型的应用可行性： 1. **信息提取**：从植物园出版的历史种子目录（种子名录）中抽取植物物种名称。 2. **自然语言理解**：从欧盟健康技术评估组织发布的文档中抽取特定的数据点，例如药物名称、健康指征名称、相对效果和成本效益等。 3. **文本分类**：为众筹网站Kickstarter上的项目分配行业代码。 我们分享了在这些应用场景中学到的经验教训：如何判断生成式AI是否适合某一具体数据处理任务，并且如果合适的话，如何最大程度地提高所得结果的准确性和一致性。
URL
https://arxiv.org/abs/2504.15829
PDF
-
LLM-based Semantic Augmentation for Harmful Content Detection
2025-04-22 02:59:03Elyas Meguellati, Assaad Zeghina, Shazia Sadiq, Gianluca DemartiniAbstract
Recent advances in large language models (LLMs) have demonstrated strong performance on simple text classification tasks, frequently under zero-shot settings. However, their efficacy declines when tackling complex social media challenges such as propaganda detection, hateful meme classification, and toxicity identification. Much of the existing work has focused on using LLMs to generate synthetic training data, overlooking the potential of LLM-based text preprocessing and semantic augmentation. In this paper, we introduce an approach that prompts LLMs to clean noisy text and provide context-rich explanations, thereby enhancing training sets without substantial increases in data volume. We systematically evaluate on the SemEval 2024 multi-label Persuasive Meme dataset and further validate on the Google Jigsaw toxic comments and Facebook hateful memes datasets to assess generalizability. Our results reveal that zero-shot LLM classification underperforms on these high-context tasks compared to supervised models. In contrast, integrating LLM-based semantic augmentation yields performance on par with approaches that rely on human-annotated data, at a fraction of the cost. These findings underscore the importance of strategically incorporating LLMs into machine learning (ML) pipeline for social media classification tasks, offering broad implications for combating harmful content online.
Abstract (translated)
最近在大型语言模型（LLMs）方面取得的进展表明，这些模型在简单的文本分类任务上表现出色，并且通常可以在零样本设置下完成任务。然而，在处理复杂的社交媒体挑战如宣传检测、仇恨言论识别和毒性内容鉴定等情境时，它们的效果会显著下降。目前大部分研究都集中在使用LLM生成合成训练数据上，而忽略了基于LLM的文本预处理和语义增强的潜力。 在本文中，我们提出了一种新方法：通过提示LLM来清理嘈杂文本并提供丰富的背景解释，以此在不大幅增加数据量的情况下改进训练集。我们在SemEval 2024多标签说服性表情包（Persuasive Meme）数据集中进行了系统的评估，并进一步在谷歌Jigsaw有毒评论和Facebook仇恨言论表情包的数据集上进行验证以测试其泛化能力。我们的结果显示，零样本LLM分类在此类高上下文任务中表现不如监督模型。相比之下，结合基于LLM的语义增强方法，则可以达到与依赖人工标注数据的方法相媲美的性能，并且成本更低。 这些研究结果强调了在社交媒体分类任务中战略性的将大型语言模型集成到机器学习（ML）流程中的重要性，并为打击网上的有害内容提供了广泛的应用前景。
URL
https://arxiv.org/abs/2504.15548
PDF
-
Feeding LLM Annotations to BERT Classifiers at Your Own Risk
2025-04-21 20:54:55Yucheng Lu, Kazimier SmithAbstract
Using LLM-generated labels to fine-tune smaller encoder-only models for text classification has gained popularity in various settings. While this approach may be justified in simple and low-stakes applications, we conduct empirical analysis to demonstrate how the perennial curse of training on synthetic data manifests itself in this specific setup. Compared to models trained on gold labels, we observe not only the expected performance degradation in accuracy and F1 score, but also increased instability across training runs and premature performance plateaus. These findings cast doubts on the reliability of such approaches in real-world applications. We contextualize the observed phenomena through the lens of error propagation and offer several practical mitigation strategies, including entropy-based filtering and ensemble techniques. Although these heuristics offer partial relief, they do not fully resolve the inherent risks of propagating non-random errors from LLM annotations to smaller classifiers, underscoring the need for caution when applying this workflow in high-stakes text classification tasks.
Abstract (translated)
使用大型语言模型（LLM）生成的标签来微调较小的编码器-only模型进行文本分类，在各种场景中越来越受欢迎。尽管这种方法在简单和低风险的应用程序中可能是合理的，但我们通过实证分析展示了在这种特定设置下训练时使用合成数据所带来的长期困扰如何体现出来。与基于黄金标准标签训练的模型相比，我们不仅观察到了预期中的准确率和F1分数下降，还发现了跨训练运行的不稳定性增加以及性能过早达到饱和的现象。这些发现对这种方法在现实世界应用中的可靠性提出了质疑。 我们通过错误传播的角度来解释所观察到的现象，并提出了一些实用的缓解策略，包括基于熵的过滤方法和集成技术。尽管这些经验法则提供了一定程度上的缓解效果，但它们并不能完全解决将非随机错误从LLM注释传递给较小分类器的根本风险问题，这强调了在高风险文本分类任务中应用这一工作流程时需谨慎行事的重要性。
URL
https://arxiv.org/abs/2504.15432
PDF
-
Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimization
2025-04-19 06:26:23Huiyi Chen, Jiawei Peng, Kaihua Tang, Xin Geng, Xu YangAbstract
In-context learning (ICL) enables Large Vision-Language Models (LVLMs) to adapt to new tasks without parameter updates, using a few demonstrations from a large support set. However, selecting informative demonstrations leads to high computational and memory costs. While some methods explore selecting a small and representative coreset in the text classification, evaluating all support set samples remains costly, and discarded samples lead to unnecessary information loss. These methods may also be less effective for image classification due to differences in feature spaces. Given these limitations, we propose Key-based Coreset Optimization (KeCO), a novel framework that leverages untapped data to construct a compact and informative coreset. We introduce visual features as keys within the coreset, which serve as the anchor for identifying samples to be updated through different selection strategies. By leveraging untapped samples from the support set, we update the keys of selected coreset samples, enabling the randomly initialized coreset to evolve into a more informative coreset under low computational cost. Through extensive experiments on coarse-grained and fine-grained image classification benchmarks, we demonstrate that KeCO effectively enhances ICL performance for image classification task, achieving an average improvement of more than 20\%. Notably, we evaluate KeCO under a simulated online scenario, and the strong performance in this scenario highlights the practical value of our framework for resource-constrained real-world scenarios.
Abstract (translated)
上下文学习（ICL）使大型视觉-语言模型（LVLMs）能够在不更新参数的情况下适应新任务，通过从一个大的支持集合中使用少量的演示案例来实现这一点。然而，选择具有信息性的演示案例会导致计算和内存成本高昂。虽然有一些方法探索了在文本分类中选取小而有代表性的核心集的方法，但评估所有支持集合样本仍然代价高昂，并且被丢弃的样本会导致不必要的信息损失。这些方法也可能因为特征空间的不同而在图像分类任务上效果较差。 鉴于上述限制，我们提出了一种基于键的关键核心优化（KeCO）框架，该框架利用未充分利用的数据来构建一个紧凑而有信息量的核心集。我们在核心集中引入了视觉特征作为“键”，用作通过不同选择策略识别需要更新的样本的基础。通过利用支持集合中未使用的样本，我们更新选定核心集样本中的“键”，从而使随机初始化的核心集在较低计算成本下演变为更有信息量的核心集。 通过在粗粒度和细粒度图像分类基准上的广泛实验，我们证明了KeCO能够有效地提高ICL在图像分类任务上的性能，平均提高了超过20%。值得注意的是，我们在模拟的在线场景中评估了KeCO，并且该框架在此场景中的出色表现突显了它对于资源受限的真实世界应用场景的实际价值。
URL
https://arxiv.org/abs/2504.14200
PDF
-
Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
2025-04-18 17:54:33Junjie Yang, Junhao Song, Xudong Han, Ziqian Bi, Tianyang Wang, Chia Xin Liang, Xinyuan Song, Yichao Zhang, Qian Niu, Benji Peng, Keyu Chen, Ming LiuAbstract
Knowledge distillation (KD) is a technique for transferring knowledge from complex teacher models to simpler student models, significantly enhancing model efficiency and accuracy. It has demonstrated substantial advancements in various applications including image classification, object detection, language modeling, text classification, and sentiment analysis. Recent innovations in KD methods, such as attention-based approaches, block-wise logit distillation, and decoupling distillation, have notably improved student model performance. These techniques focus on stimulus complexity, attention mechanisms, and global information capture to optimize knowledge transfer. In addition, KD has proven effective in compressing large language models while preserving accuracy, reducing computational overhead, and improving inference speed. This survey synthesizes the latest literature, highlighting key findings, contributions, and future directions in knowledge distillation to provide insights for researchers and practitioners on its evolving role in artificial intelligence and machine learning.
Abstract (translated)
知识蒸馏（KD）是一种将复杂教师模型的知识转移到更简单的学生模型的技术，显著提升了模型的效率和准确性。它在图像分类、目标检测、语言建模、文本分类以及情感分析等多种应用中取得了重要进展。最近关于KD方法的创新，例如基于注意力的方法、分块对数蒸馏（block-wise logit distillation）及解耦蒸馏（decoupling distillation），显著提高了学生模型的表现力。这些技术着重于刺激复杂度、注意机制以及全局信息捕获，以优化知识转移过程。此外，KD在压缩大型语言模型的同时保持准确性，在减少计算开销和提高推理速度方面表现出色。 本文综述了最新的研究文献，重点介绍了知识蒸馏领域的关键发现、贡献及未来发展方向，为研究人员与实践者提供了关于其在人工智能及机器学习领域内不断演变作用的见解。
URL
https://arxiv.org/abs/2504.13825
PDF
-
Word Embedding Techniques for Classification of Star Ratings
2025-04-18 12:26:28Hesham Abdelmotaleb, Craig McNeile, Malgorzata WojtysAbstract
Telecom services are at the core of today's societies' everyday needs. The availability of numerous online forums and discussion platforms enables telecom providers to improve their services by exploring the views of their customers to learn about common issues that the customers face. Natural Language Processing (NLP) tools can be used to process the free text collected. One way of working with such data is to represent text as numerical vectors using one of many word embedding models based on neural networks. This research uses a novel dataset of telecom customers' reviews to perform an extensive study showing how different word embedding algorithms can affect the text classification process. Several state-of-the-art word embedding techniques are considered, including BERT, Word2Vec and Doc2Vec, coupled with several classification algorithms. The important issue of feature engineering and dimensionality reduction is addressed and several PCA-based approaches are explored. Moreover, the energy consumption used by the different word embeddings is investigated. The findings show that some word embedding models can lead to consistently better text classifiers in terms of precision, recall and F1-Score. In particular, for the more challenging classification tasks, BERT combined with PCA stood out with the highest performance metrics. Moreover, our proposed PCA approach of combining word vectors using the first principal component shows clear advantages in performance over the traditional approach of taking the average.
Abstract (translated)
电信服务是当今社会日常需求的核心。众多在线论坛和讨论平台的可用性使电信提供商能够通过探索客户意见来改进其服务，从而了解客户面临的常见问题。自然语言处理（NLP）工具可以用来处理收集到的自由文本。处理此类数据的一种方法是使用基于神经网络的多种词嵌入模型之一将文本表示为数值向量。这项研究利用了一套新颖的电信客户评论数据集，对不同词嵌入算法如何影响文本分类过程进行了详尽的研究。考虑了包括BERT、Word2Vec和Doc2Vec在内的几种最先进的词嵌入技术，并结合了几种分类算法。此外还探讨了特征工程和降维这一重要问题，并探索了几种基于PCA的方法。另外，这项研究还调查了不同词嵌入模型的能耗情况。 研究表明，某些词嵌入模型可以导致在精度、召回率和F1分数方面表现更好的文本分类器。特别是对于更具挑战性的分类任务，BERT与PCA结合使用时表现出最高的性能指标。此外，我们提出的基于PCA的方法，即利用第一主成分组合词向量，在性能上明显优于传统的平均方法。
URL
https://arxiv.org/abs/2504.13653
PDF
-
Selective Attention Federated Learning: Improving Privacy and Efficiency for Clinical Text Classification
2025-04-16 05:59:29Yue Li, Lihong ZhangAbstract
Federated Learning (FL) faces major challenges regarding communication overhead and model privacy when training large language models (LLMs), especially in healthcare applications. To address these, we introduce Selective Attention Federated Learning (SAFL), a novel approach that dynamically fine-tunes only those transformer layers identified as attention-critical. By employing attention patterns to determine layer importance, SAFL significantly reduces communication bandwidth and enhances differential privacy resilience. Evaluations on clinical NLP benchmarks (i2b2 Clinical Concept Extraction and MIMIC-III discharge summaries) demonstrate that SAFL achieves competitive performance with centralized models while substantially improving communication efficiency and privacy preservation.
Abstract (translated)
联邦学习（FL）在训练大型语言模型（LLMs）时，特别是在医疗应用中，面临着通信开销和模型隐私的重大挑战。为了解决这些问题，我们引入了一种新颖的方法——选择性注意力联邦学习（SAFL），该方法动态地只对那些被识别为关键注意层的变换器层进行微调。通过利用注意力模式来确定各层的重要性，SAFL显著减少了通信带宽，并增强了差分隐私的抵抗能力。在临床自然语言处理基准测试（i2b2临床概念抽取和MIMIC-III出院总结）上的评估表明，SAFL能够在保持与集中式模型相当性能的同时，大幅提高通信效率并保护隐私。
URL
https://arxiv.org/abs/2504.11793
PDF
-
Langformers: Unified NLP Pipelines for Language Models
2025-04-12 10:17:49Rabindra Lamsal, Maria Rodriguez Read, Shanika KarunasekeraAbstract
Transformer-based language models have revolutionized the field of natural language processing (NLP). However, using these models often involves navigating multiple frameworks and tools, as well as writing repetitive boilerplate code. This complexity can discourage non-programmers and beginners, and even slow down prototyping for experienced developers. To address these challenges, we introduce Langformers, an open-source Python library designed to streamline NLP pipelines through a unified, factory-based interface for large language model (LLM) and masked language model (MLM) tasks. Langformers integrates conversational AI, MLM pretraining, text classification, sentence embedding/reranking, data labelling, semantic search, and knowledge distillation into a cohesive API, supporting popular platforms such as Hugging Face and Ollama. Key innovations include: (1) task-specific factories that abstract training, inference, and deployment complexities; (2) built-in memory and streaming for conversational agents; and (3) lightweight, modular design that prioritizes ease of use. Documentation: this https URL
Abstract (translated)
基于Transformer的语言模型已经彻底革新了自然语言处理（NLP）领域。然而，使用这些模型通常涉及到在多个框架和工具之间切换，并且需要编写大量的样板代码。这种复杂性可能会让非编程人员和初学者望而却步，甚至会减慢经验丰富的开发者的原型设计过程。为了应对这些挑战，我们推出了Langformers——一个开源的Python库，旨在通过统一、工厂式的界面简化NLP流水线，该界面支持大型语言模型（LLM）和掩码语言模型（MLM）任务。 Langformers将对话AI、MLM预训练、文本分类、句子嵌入/重排序、数据标注、语义搜索以及知识蒸馏等整合到一个连贯的API中，并且它支持流行平台如Hugging Face和Ollama。关键创新包括： 1. 任务特定工厂：抽象了训练、推理和部署的复杂性。 2. 内置记忆与流处理功能，专为对话代理设计。 3. 轻量级模块化设计，优先考虑易用性。 文档：[此处插入具体URL链接]
URL
https://arxiv.org/abs/2504.09170
PDF
-
Integrated ensemble of BERT- and features-based models for authorship attribution in Japanese literary works
2025-04-11 13:40:50Taisei Kanda, Mingzhe Jin, Wataru ZaitsuAbstract
Traditionally, authorship attribution (AA) tasks relied on statistical data analysis and classification based on stylistic features extracted from texts. In recent years, pre-trained language models (PLMs) have attracted significant attention in text classification tasks. However, although they demonstrate excellent performance on large-scale short-text datasets, their effectiveness remains under-explored for small samples, particularly in AA tasks. Additionally, a key challenge is how to effectively leverage PLMs in conjunction with traditional feature-based methods to advance AA research. In this study, we aimed to significantly improve performance using an integrated integrative ensemble of traditional feature-based and modern PLM-based methods on an AA task in a small sample. For the experiment, we used two corpora of literary works to classify 10 authors each. The results indicate that BERT is effective, even for small-sample AA tasks. Both BERT-based and classifier ensembles outperformed their respective stand-alone models, and the integrated ensemble approach further improved the scores significantly. For the corpus that was not included in the pre-training data, the integrated ensemble improved the F1 score by approximately 14 points, compared to the best-performing single model. Our methodology provides a viable solution for the efficient use of the ever-expanding array of data processing tools in the foreseeable future.
Abstract (translated)
传统上，作者身份归属（AA）任务依赖于基于从文本中提取的风格特征进行统计数据分析和分类。近年来，在文本分类任务中，预训练语言模型（PLM）引起了广泛关注。然而，尽管它们在大规模短文本数据集上的表现优异，但对于小样本尤其是在AA任务中的有效性仍需进一步探索。此外，一个关键挑战是如何有效结合传统基于特征的方法与现代PLM方法来推进AA研究。在这项研究中，我们旨在通过整合传统的基于特征和现代的基于PLM的方法，在小样本AA任务上显著提升性能。实验中，我们使用了两部文学作品语料库，每个语料库包含10位作者的作品。 结果表明，即使在小样本AA任务中，BERT也表现出有效性。基于BERT和分类器集成方法均优于各自独立的模型，并且综合集成的方法进一步显著提高了评分。对于未包括在预训练数据中的语料库，与表现最佳的单一模型相比，综合集成方法将F1分数提升了大约14个百分点。我们的研究方法为未来有效利用不断扩大的数据处理工具提供了可行方案。
URL
https://arxiv.org/abs/2504.08527
PDF
-
Specialized text classification: an approach to classifying Open Banking transactions
2025-04-10 17:14:43Duc Tuyen TA, Wajdi Ben Saad, Ji Young OhAbstract
With the introduction of the PSD2 regulation in the EU which established the Open Banking framework, a new window of opportunities has opened for banks and fintechs to explore and enrich Bank transaction descriptions with the aim of building a better understanding of customer behavior, while using this understanding to prevent fraud, reduce risks and offer more competitive and tailored services. And although the usage of natural language processing models and techniques has seen an incredible progress in various applications and domains over the past few years, custom applications based on domain-specific text corpus remain unaddressed especially in the banking sector. In this paper, we introduce a language-based Open Banking transaction classification system with a focus on the french market and french language text. The system encompasses data collection, labeling, preprocessing, modeling, and evaluation stages. Unlike previous studies that focus on general classification approaches, this system is specifically tailored to address the challenges posed by training a language model with a specialized text corpus (Banking data in the French context). By incorporating language-specific techniques and domain knowledge, the proposed system demonstrates enhanced performance and efficiency compared to generic approaches.
Abstract (translated)
随着欧盟引入PSD2法规，建立了开放银行框架，为银行和金融科技公司提供了一个新的机遇窗口，可以探索并丰富银行交易描述，以更好地理解客户行为。利用这种理解来预防欺诈、降低风险，并提供更具竞争力且个性化的服务。尽管近年来自然语言处理模型和技术在各种应用和领域中取得了惊人的进展，但在特定领域的文本语料库基础上开发定制应用程序方面仍然未得到充分解决，尤其是在银行业。 本文介绍了一种基于语言的开放银行交易分类系统，重点关注法国市场和法语文本。该系统涵盖了数据收集、标注、预处理、建模和评估阶段。与以往专注于通用分类方法的研究不同，本系统特别针对使用特定领域文本语料库（在法国背景下为银行业务数据）训练语言模型所面临的挑战进行了定制化设计。通过结合语言特有技术和领域知识，该提议的系统相比通用方法表现出更优的性能和效率。 这一系统的开发标志着向更精细、更具针对性的数据驱动解决方案迈出了重要一步，这将有助于银行更好地理解客户需求，并提供更加个性化且有效的服务。
URL
https://arxiv.org/abs/2504.12319
PDF
-
A new training approach for text classification in Mental Health: LatentGLoss
2025-04-09 19:34:31Korhan Sevin\c{c}Abstract
This study presents a multi-stage approach to mental health classification by leveraging traditional machine learning algorithms, deep learning architectures, and transformer-based models. A novel data set was curated and utilized to evaluate the performance of various methods, starting with conventional classifiers and advancing through neural networks. To broaden the architectural scope, recurrent neural networks (RNNs) such as LSTM and GRU were also evaluated to explore their effectiveness in modeling sequential patterns in the data. Subsequently, transformer models such as BERT were fine-tuned to assess the impact of contextual embeddings in this domain. Beyond these baseline evaluations, the core contribution of this study lies in a novel training strategy involving a dual-model architecture composed of a teacher and a student network. Unlike standard distillation techniques, this method does not rely on soft label transfer; instead, it facilitates information flow through both the teacher model's output and its latent representations by modifying the loss function. The experimental results highlight the effectiveness of each modeling stage and demonstrate that the proposed loss function and teacher-student interaction significantly enhance the model's learning capacity in mental health prediction tasks.
Abstract (translated)
这项研究提出了一种多阶段方法，用于通过利用传统机器学习算法、深度学习架构和基于变换器的模型来分类心理健康。研究人员精心策划并使用了一个新颖的数据集，以评估各种方法的性能，从传统的分类器开始，并逐步过渡到神经网络。为了拓宽架构范围，还评估了诸如LSTM（长短期记忆）和GRU（门控循环单元）等递归神经网络，以探索它们在建模数据中序列模式方面的有效性。 随后，对BERT等变换模型进行了微调，以评估上下文嵌入在此领域的影响力。除了这些基准测试外，这项研究的核心贡献在于一种新颖的训练策略，该策略涉及由教师和学生网络组成的双模型架构。与标准蒸馏技术不同，这种方法不依赖于软标签转移；相反，通过修改损失函数来促进信息流通过教师模型的输出及其潜在表示。 实验结果突显了每个建模阶段的有效性，并展示了所提出的损失函数及师生互动显著增强了心理健康预测任务中模型的学习能力。
URL
https://arxiv.org/abs/2504.07245
PDF
-
Linguistic Interpretability of Transformer-based Language Models: a systematic review
2025-04-09 08:00:12Miguel L\'opez-Otal, Jorge Gracia, Jordi Bernad, Carlos Bobed, Luc\'ia Pitarch-Ballesteros, Emma Angl\'es-HerreroAbstract
Language models based on the Transformer architecture achieve excellent results in many language-related tasks, such as text classification or sentiment analysis. However, despite the architecture of these models being well-defined, little is known about how their internal computations help them achieve their results. This renders these models, as of today, a type of 'black box' systems. There is, however, a line of research -- 'interpretability' -- aiming to learn how information is encoded inside these models. More specifically, there is work dedicated to studying whether Transformer-based models possess knowledge of linguistic phenomena similar to human speakers -- an area we call 'linguistic interpretability' of these models. In this survey we present a comprehensive analysis of 160 research works, spread across multiple languages and models -- including multilingual ones -- that attempt to discover linguistic information from the perspective of several traditional Linguistics disciplines: Syntax, Morphology, Lexico-Semantics and Discourse. Our survey fills a gap in the existing interpretability literature, which either not focus on linguistic knowledge in these models or present some limitations -- e.g. only studying English-based models. Our survey also focuses on Pre-trained Language Models not further specialized for a downstream task, with an emphasis on works that use interpretability techniques that explore models' internal representations.
Abstract (translated)
基于Transformer架构的语言模型在许多与语言相关的任务（如文本分类或情感分析）中取得了出色的成绩。然而，尽管这些模型的架构已经被很好地定义了，但对于它们内部计算如何帮助其取得结果知之甚少。这使得当前的这些模型成为一种“黑箱”系统。然而，有一条研究路线——“可解释性”，旨在学习这些模型内部信息是如何编码的。具体来说，有些工作致力于研究基于Transformer的模型是否拥有类似于人类语言使用者的语言现象知识——我们称这一领域为“这些模型的语言可解释性”。在这篇综述中，我们对160项跨多种语言和模型（包括多语种模型）的研究进行了全面分析，试图从传统语言学领域的多个视角（如句法、形态学、词汇-语义和话语）发现这些模型中的语言信息。我们的调查填补了现有可解释性文献的一个空白，即要么不关注这些模型中的语言知识，要么存在一些局限——例如只研究基于英语的模型。此外，本综述还侧重于未进一步针对下游任务进行专门化的预训练语言模型，并强调使用探索模型内部表示的技术来实现可解释性的作品。
URL
https://arxiv.org/abs/2504.08001
PDF
-
Layer-Aware Embedding Fusion for LLMs in Text Classifications
2025-04-08 07:45:50Jiho Gwak, Yuchul JungAbstract
Embedding fusion has emerged as an effective approach for enhancing performance across various NLP tasks. However, systematic guidelines for selecting optimal layers and developing effective fusion strategies for the integration of LLMs remain underexplored. In this study, we propose a layer-aware embedding selection method and investigate how to quantitatively evaluate different layers to identify the most important ones for downstream NLP tasks, showing that the critical layers vary depending on the dataset. We also explore how combining embeddings from multiple LLMs, without requiring model fine-tuning, can improve performance. Experiments on four English text classification datasets (SST-2, MR, R8, and R52) demonstrate that different layers in LLMs exhibit varying degrees of representational strength for classification, and that combining embeddings from different models can enhance performance if the models exhibit complementary characteristics. Additionally, we discuss resources overhead (memory and inference time) to provide a balanced perspective on the real world feasibility of embedding fusion. Future work will explore multilingual and domain specific datasets, as well as techniques for automating layer selection, to improve both performance and scalability.
Abstract (translated)
嵌入融合已成为增强各种自然语言处理（NLP）任务性能的有效方法。然而，关于如何选择最佳层次以及为大规模语言模型（LLMs）的集成开发有效的融合策略，系统性指导原则尚未得到充分探索。在这项研究中，我们提出了一种基于层的嵌入选择方法，并探讨了如何定量评估不同层次以识别对下游NLP任务最重要的那些层次，研究表明关键层次会因数据集的不同而变化。此外，我们还探索了在无需微调模型的情况下，通过结合来自多个LLMs的嵌入来改进性能的方法。我们在四个英文文本分类数据集（SST-2、MR、R8和R52）上的实验表明，LLMs中的不同层次对于分类任务表现出不同的表征能力，并且如果这些模型具有互补特征，则结合来自不同模型的嵌入可以提高性能。我们还讨论了资源开销（内存和推理时间），以提供关于嵌入融合在现实世界可行性方面的平衡视角。未来的研究将探索多语言及特定领域的数据集，以及自动选择层次的技术，从而进一步提升性能和可扩展性。
URL
https://arxiv.org/abs/2504.05764
PDF
-
Batch Aggregation: An Approach to Enhance Text Classification with Correlated Augmented Data
2025-04-07 12:46:07Charco Hui, Yalu WenAbstract
Natural language processing models often face challenges due to limited labeled data, especially in domain specific areas, e.g., clinical trials. To overcome this, text augmentation techniques are commonly used to increases sample size by transforming the original input data into artificial ones with the label preserved. However, traditional text classification methods ignores the relationship between augmented texts and treats them as independent samples which may introduce classification error. Therefore, we propose a novel approach called 'Batch Aggregation' (BAGG) which explicitly models the dependence of text inputs generated through augmentation by incorporating an additional layer that aggregates results from correlated texts. Through studying multiple benchmark data sets across different domains, we found that BAGG can improve classification accuracy. We also found that the increase of performance with BAGG is more obvious in domain specific data sets, with accuracy improvements of up to 10-29%. Through the analysis of benchmark data, the proposed method addresses limitations of traditional techniques and improves robustness in text classification tasks. Our result demonstrates that BAGG offers more robust results and outperforms traditional approaches when training data is limited.
Abstract (translated)
自然语言处理模型常因标注数据有限而在特定领域（如临床试验）中面临挑战。为解决这一问题，通常采用文本增强技术来通过转换原始输入数据生成人工样本以增加样本量，并保留原有标签信息。然而，传统文本分类方法忽略了这些增强文本之间的关系，将其视为独立的样本，这可能会引入分类错误。为此，我们提出了一种名为“批量聚合”（Batch Aggregation，简称BAGG）的新方法，该方法通过添加一层来明确建模通过增强生成的文本输入间的关系，这一层用于聚集相关文本的结果。 通过对不同领域内的多个基准数据集的研究，我们发现BAGG可以提高分类准确率。此外，在特定领域的数据集中，使用BAGG性能提升更为显著，最高可达到10-29%的精度改善。通过分析基准数据，所提出的方法解决了传统技术的局限性，并增强了文本分类任务中的鲁棒性。 我们的结果显示，在训练数据有限的情况下，BAGG方法提供更稳健的结果并优于传统的做法。
URL
https://arxiv.org/abs/2504.05020
PDF
-
GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models
2025-04-05 12:30:58Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Ra\'ul V\'azquez, Mengjie Wang, Samea Yusofi, J\"org TiedemannAbstract
Large language models (LLMs) are advancing at an unprecedented pace globally, with regions increasingly adopting these models for applications in their primary language. Evaluation of these models in diverse linguistic environments, especially in low-resource languages, has become a major challenge for academia and industry. Existing evaluation frameworks are disproportionately focused on English and a handful of high-resource languages, thereby overlooking the realistic performance of LLMs in multilingual and lower-resource scenarios. To address this gap, we introduce GlotEval, a lightweight framework designed for massively multilingual evaluation. Supporting seven key tasks (machine translation, text classification, summarization, open-ended generation, reading comprehension, sequence labeling, and intrinsic evaluation), spanning over dozens to hundreds of languages, GlotEval highlights consistent multilingual benchmarking, language-specific prompt templates, and non-English-centric machine translation. This enables a precise diagnosis of model strengths and weaknesses in diverse linguistic contexts. A multilingual translation case study demonstrates GlotEval's applicability for multilingual and language-specific evaluations.
Abstract (translated)
大型语言模型（LLMs）在全球范围内以前所未有的速度发展，各地区越来越多地采用这些模型来支持其主要语言的应用。在多语种环境中特别是低资源语言中评估这些模型已成为学术界和工业界的巨大挑战。现有的评估框架过于侧重于英语和其他几种高资源语言，从而忽略了大规模多语言及低资源场景下大型语言模型的真实性能表现。为解决这一问题，我们引入了GlotEval，这是一个轻量级的框架，旨在进行大规模多语种评估。该框架支持七项关键任务（机器翻译、文本分类、摘要生成、开放性生成、阅读理解、序列标注和内在评测），覆盖数十到数百种语言，强调一致性的跨语言基准测试、语言特定的提示模板以及非英语中心化的机器翻译。这使得能够精确诊断模型在不同语言环境中的优势与劣势。 一个多语种翻译案例研究展示了GlotEval在多语言及语言特有评估方面的适用性。
URL
https://arxiv.org/abs/2504.04155
PDF
-
Beyond the Next Token: Towards Prompt-Robust Zero-Shot Classification via Efficient Multi-Token Prediction
2025-04-04 04:39:51Junlang Qian, Zixiao Zhu, Hanzhang Zhou, Zijian Feng, Zepeng Zhai, Kezhi MaoAbstract
Zero-shot text classification typically relies on prompt engineering, but the inherent prompt brittleness of large language models undermines its reliability. Minor changes in prompt can cause significant discrepancies in model performance. We attribute this prompt brittleness largely to the narrow focus on nexttoken probabilities in existing methods. To address this, we propose Placeholding Parallel Prediction (P3), a novel approach that predicts token probabilities across multiple positions and simulates comprehensive sampling of generation paths in a single run of a language model. Experiments show improved accuracy and up to 98% reduction in the standard deviation across prompts, boosting robustness. Even without a prompt, P3 maintains comparable performance, reducing the need for prompt engineering.
Abstract (translated)
零样本文本分类通常依赖于提示工程，但大型语言模型内在的提示脆弱性影响了其可靠性。即使是微小的提示变化也可能导致模型性能出现显著差异。我们认为这种提示脆弱性的主要原因是现有方法对下一个词概率的狭窄关注。为解决这一问题，我们提出了一种新颖的方法——占位并行预测（Placeholding Parallel Prediction, P3），该方法可以同时在多个位置预测词的概率，并在一个语言模型运行过程中模拟生成路径的全面采样。实验表明，P3 方法不仅提高了准确性，还使提示间的标准差降低了高达 98%，从而增强了模型的稳健性。即使没有提示的情况下，P3 方法也能保持相当的性能水平，减少了对复杂提示工程的需求。
URL
https://arxiv.org/abs/2504.03159
PDF
-
A thorough benchmark of automatic text classification: From traditional approaches to large language models
2025-04-02 17:40:08Washington Cunha, Leonardo Rocha, Marcos Andr\'e Gon\c{c}alvesAbstract
Automatic text classification (ATC) has experienced remarkable advancements in the past decade, best exemplified by recent small and large language models (SLMs and LLMs), leveraged by Transformer architectures. Despite recent effectiveness improvements, a comprehensive cost-benefit analysis investigating whether the effectiveness gains of these recent approaches compensate their much higher costs when compared to more traditional text classification approaches such as SVMs and Logistic Regression is still missing in the literature. In this context, this work's main contributions are twofold: (i) we provide a scientifically sound comparative analysis of the cost-benefit of twelve traditional and recent ATC solutions including five open LLMs, and (ii) a large benchmark comprising {22 datasets}, including sentiment analysis and topic classification, with their (train-validation-test) partitions based on folded cross-validation procedures, along with documentation, and code. The release of code, data, and documentation enables the community to replicate experiments and advance the field in a more scientifically sound manner. Our comparative experimental results indicate that LLMs outperform traditional approaches (up to 26%-7.1% on average) and SLMs (up to 4.9%-1.9% on average) in terms of effectiveness. However, LLMs incur significantly higher computational costs due to fine-tuning, being, on average 590x and 8.5x slower than traditional methods and SLMs, respectively. Results suggests the following recommendations: (1) LLMs for applications that require the best possible effectiveness and can afford the costs; (2) traditional methods such as Logistic Regression and SVM for resource-limited applications or those that cannot afford the cost of tuning large LLMs; and (3) SLMs like Roberta for near-optimal effectiveness-efficiency trade-off.
Abstract (translated)
自动文本分类（ATC）在过去十年中取得了显著的进展，这一进步最典型的例子是近期的小型和大型语言模型（SLMs 和 LLMs），这些模型采用了Transformer架构。尽管最近的有效性有了很大的提高，但文献中仍然缺乏一种全面的成本效益分析来研究这些新方法的效果提升是否能够补偿其相比于传统的文本分类方法如支持向量机(SVM)和逻辑回归等所增加的高昂成本。 在这个背景下，这项工作的主要贡献包括两个方面：(i) 我们提供了一种科学合理的比较分析，评估了十二种传统和近期ATC解决方案的成本效益，其中包括五种开源大型语言模型；(ii) 一个大规模基准测试集合，包含22个数据集（如情感分析和主题分类），这些数据集基于折叠交叉验证过程划分的训练、验证和测试部分，并附有文档和代码。代码、数据及文档的发布使得社区能够复制实验并以更科学的方式推进领域研究。 我们的比较实验结果表明，在有效性方面，大型语言模型(LLMs)优于传统方法（平均提升26%-7.1%）和小型语言模型(SLMs)（平均提升4.9%-1.9%）。然而，LLMs由于需要进行微调而产生显著更高的计算成本，与传统方法相比平均慢590倍，比SLMs慢8.5倍。实验结果表明以下建议：(1) 对于那些要求最佳性能且能承担相应费用的应用程序，推荐使用大型语言模型；(2) 资源有限或无法承受微调大LLM成本的场景，则适合采用传统方法如逻辑回归和SVM；(3) 需要接近最优效率-效果平衡的应用场景可以考虑选择Roberta等小型语言模型。
URL
https://arxiv.org/abs/2504.01930
PDF
-
A Novel Approach To Implementing Knowledge Distillation In Tsetlin Machines
2025-04-02 15:06:27Calvin KinatederAbstract
The Tsetlin Machine (TM) is a propositional logic based model that uses conjunctive clauses to learn patterns from data. As with typical neural networks, the performance of a Tsetlin Machine is largely dependent on its parameter count, with a larger number of parameters producing higher accuracy but slower execution. Knowledge distillation in neural networks transfers information from an already-trained teacher model to a smaller student model to increase accuracy in the student without increasing execution time. We propose a novel approach to implementing knowledge distillation in Tsetlin Machines by utilizing the probability distributions of each output sample in the teacher to provide additional context to the student. Additionally, we propose a novel clause-transfer algorithm that weighs the importance of each clause in the teacher and initializes the student with only the most essential data. We find that our algorithm can significantly improve performance in the student model without negatively impacting latency in the tested domains of image recognition and text classification.
Abstract (translated)
Tsetlin 机（TM）是一种基于命题逻辑的模型，它使用合取子句从数据中学习模式。与典型的神经网络一样，Tsetlin 机器的性能在很大程度上取决于其参数数量：更多的参数可以提高准确率，但会导致执行速度变慢。知识蒸馏技术通常是在已训练好的教师模型和较小的学生模型之间转移信息，以此增加学生模型的准确性而不延长执行时间。 我们提出了一种新的方法，在 Tsetlin 机器中实施知识蒸馏，通过利用教师模型每个输出样本的概率分布向学生提供额外的信息上下文。此外，我们还提出了一个新颖的子句传输算法，该算法根据其在教师中的重要性对每个子句进行加权，并且只用最关键的数据初始化学生模型。 我们的实验结果表明，在图像识别和文本分类等测试领域中，本算法可以在不增加延迟的情况下显著提高学生模型的表现。
URL
https://arxiv.org/abs/2504.01798
PDF
-
WHERE and WHICH: Iterative Debate for Biomedical Synthetic Data Augmentation
2025-03-31 02:36:30Zhengyi Zhao, Shubo Zhang, Bin Liang, Binyang Li, Kam-Fai WongAbstract
In Biomedical Natural Language Processing (BioNLP) tasks, such as Relation Extraction, Named Entity Recognition, and Text Classification, the scarcity of high-quality data remains a significant challenge. This limitation poisons large language models to correctly understand relationships between biological entities, such as molecules and diseases, or drug interactions, and further results in potential misinterpretation of biomedical documents. To address this issue, current approaches generally adopt the Synthetic Data Augmentation method which involves similarity computation followed by word replacement, but counterfactual data are usually generated. As a result, these methods disrupt meaningful word sets or produce sentences with meanings that deviate substantially from the original context, rendering them ineffective in improving model performance. To this end, this paper proposes a biomedical-dedicated rationale-based synthetic data augmentation method. Beyond the naive lexicon similarity, specific bio-relation similarity is measured to hold the augmented instance having a strong correlation with bio-relation instead of simply increasing the diversity of augmented data. Moreover, a multi-agents-involved reflection mechanism helps the model iteratively distinguish different usage of similar entities to escape falling into the mis-replace trap. We evaluate our method on the BLURB and BigBIO benchmark, which includes 9 common datasets spanning four major BioNLP tasks. Our experimental results demonstrate consistent performance improvements across all tasks, highlighting the effectiveness of our approach in addressing the challenges associated with data scarcity and enhancing the overall performance of biomedical NLP models.
Abstract (translated)
在生物医学自然语言处理（BioNLP）任务中，如关系抽取、命名实体识别和文本分类等，高质量数据的稀缺仍然是一个重大挑战。这一限制使得大型语言模型难以准确理解诸如分子与疾病之间或药物相互作用之间的生物学实体间的关系，并可能导致对生物医学文献的潜在误解。为解决此问题，目前的方法通常采用合成数据增强方法，包括相似性计算后进行词汇替换，但往往生成的是反事实数据。结果是这些方法破坏了有意义的词组集，或者产生了与原始语境显著不同的句子含义，从而无法有效提升模型性能。 为此，本文提出了一种基于生物医学专门理据的合成数据增强方法。该方法超越了简单的词汇相似度，在衡量特定生物关系的相似性时，确保增强后的实例具有较强的生物关系相关性，而不是简单地增加增强数据的多样性。此外，一种涉及多代理参与的反思机制帮助模型迭代区分类似实体的不同用法，从而避免落入错误替换的陷阱。 我们在BLURB和BigBIO基准测试上评估了我们的方法，这些基准包括涵盖四大类BioNLP任务的九个常见数据集。实验结果表明，在所有任务中均实现了持续的性能改进，这凸显了我们方法在应对数据稀缺挑战方面的有效性，并显著提升了生物医学自然语言处理模型的整体表现。
URL
https://arxiv.org/abs/2503.23673
PDF
-
Fine-Tuning LLMs on Small Medical Datasets: Text Classification and Normalization Effectiveness on Cardiology reports and Discharge records
2025-03-27 10:35:56Noah Losch, Lucas Plagwitz, Antonius B\"uscher, Julian VargheseAbstract
We investigate the effectiveness of fine-tuning large language models (LLMs) on small medical datasets for text classification and named entity recognition tasks. Using a German cardiology report dataset and the i2b2 Smoking Challenge dataset, we demonstrate that fine-tuning small LLMs locally on limited training data can improve performance achieving comparable results to larger models. Our experiments show that fine-tuning improves performance on both tasks, with notable gains observed with as few as 200-300 training examples. Overall, the study highlights the potential of task-specific fine-tuning of LLMs for automating clinical workflows and efficiently extracting structured data from unstructured medical text.
Abstract (translated)
我们研究了在小型医疗数据集上对大型语言模型（LLMs）进行微调，以改进文本分类和命名实体识别任务的有效性。通过使用德国心脏病报告数据集和i2b2吸烟挑战数据集，我们展示了在有限训练数据下本地微调小规模的LLM可以提高性能，并且能达到与更大模型相当的结果。我们的实验表明，在这两种任务上进行微调都可以改善性能，即使使用少至200-300个训练样本也观察到了显著的提升效果。总体而言，该研究强调了针对特定任务对大型语言模型进行微调在自动化临床工作流程和高效提取结构化医疗文本数据方面的潜力。
URL
https://arxiv.org/abs/2503.21349
PDF
