Text_Classification
-
GLiClass: Generalist Lightweight Model for Sequence Classification Tasks
2025-08-11 06:22:25Ihor Stepanov, Mykhailo Shtopko, Dmytro Vodianytskyi, Oleksandr Lukashov, Alexander Yavorskyi, Mykyta YaroshenkoAbstract
Classification is one of the most widespread tasks in AI applications, serving often as the first step in filtering, sorting, and categorizing data. Since modern AI systems must handle large volumes of input data and early pipeline stages can propagate errors downstream, achieving high efficiency and accuracy is critical. Moreover, classification requirements can change dynamically based on user needs, necessitating models with strong zero-shot capabilities. While generative LLMs have become mainstream for zero-shot classification due to their versatility, they suffer from inconsistent instruction following and computational inefficiency. Cross-encoders, commonly used as rerankers in RAG pipelines, face a different bottleneck: they must process text-label pairs sequentially, significantly reducing efficiency with large label sets. Embedding-based approaches offer good efficiency but struggle with complex scenarios involving logical and semantic constraints. We propose GLiClass, a novel method that adapts the GLiNER architecture for sequence classification tasks. Our approach achieves strong accuracy and efficiency comparable to embedding-based methods, while maintaining the flexibility needed for zero-shot and few-shot learning scenarios. Additionally, we adapted proximal policy optimization (PPO) for multi-label text classification, enabling training classifiers in data-sparse conditions or from human feedback.
Abstract (translated)
分类是人工智能应用中最常见的任务之一，通常作为过滤、排序和对数据进行分类的第一步。鉴于现代AI系统必须处理大量输入数据，并且早期管道阶段的错误可能会向下游传播，因此实现高效率和准确性至关重要。此外，由于用户需求的变化，分类要求可能动态变化，这需要具备强大零样本学习能力的模型。虽然基于生成的大语言模型（LLM）因其多功能性而在零样本分类中变得主流，但它们在遵循指令的一致性和计算效率方面存在问题。交叉编码器通常用于检索增强生成（RAG）管道中的重排序阶段，面临不同的瓶颈：它们必须按顺序处理文本-标签对，在面对大型标签集时会显著降低效率。基于嵌入的方法提供了较好的效率，但在涉及逻辑和语义约束的复杂场景中表现不佳。 我们提出了一种新的方法GLiClass，它将GLiNER架构应用于序列分类任务。我们的方法在准确性方面表现出色，并且与基于嵌入的方法相比，在效率方面也具有可比性，同时保持了零样本学习和少样本学习场景所需的灵活性。此外，我们将近端策略优化（PPO）用于多标签文本分类，从而能够在数据稀疏条件下或从人类反馈中训练分类器。
URL
https://arxiv.org/abs/2508.07662
PDF
-
Multi-task Adversarial Attacks against Black-box Model with Few-shot Queries
2025-08-10 12:46:47Wenqiang Wang, Yan Xiao, Hao Lin, Yangshijie Zhang, Xiaochun CaoAbstract
Current multi-task adversarial text attacks rely on abundant access to shared internal features and numerous queries, often limited to a single task type. As a result, these attacks are less effective against practical scenarios involving black-box feedback APIs, limited queries, or multiple task types. To bridge this gap, we propose \textbf{C}luster and \textbf{E}nsemble \textbf{M}ulti-task Text Adversarial \textbf{A}ttack (\textbf{CEMA}), an effective black-box attack that exploits the transferability of adversarial texts across different tasks. CEMA simplifies complex multi-task scenarios by using a \textit{deep-level substitute model} trained in a \textit{plug-and-play} manner for text classification, enabling attacks without mimicking the victim model. This approach requires only a few queries for training, converting multi-task attacks into classification attacks and allowing attacks across various tasks. CEMA generates multiple adversarial candidates using different text classification methods and selects the one that most effectively attacks substitute models. In experiments involving multi-task models with two, three, or six tasks--spanning classification, translation, summarization, and text-to-image generation--CEMA demonstrates significant attack success with as few as 100 queries. Furthermore, CEMA can target commercial APIs (e.g., Baidu and Google Translate), large language models (e.g., ChatGPT 4o), and image-generation models (e.g., Stable Diffusion V2), showcasing its versatility and effectiveness in real-world applications.
Abstract (translated)
目前的多任务对抗性文本攻击依赖于对共享内部特征的广泛访问和大量的查询，这些条件通常仅限于单一的任务类型。因此，在涉及黑盒反馈API、限制查询次数或多种任务类型的实用场景中，这类攻击的效果往往不尽如人意。为了弥合这一差距，我们提出了一种新的方法——集群与集成多任务文本对抗攻击（Cluster and Ensemble Multi-task Text Adversarial Attack, CEMA）。CEMA是一种有效的黑盒攻击策略，它利用了对抗性文本在不同任务间的可迁移性。 CEMA通过使用一种以“即插即用”方式训练的深度替代模型来简化复杂的多任务场景。这种替代模型用于文本分类，并且无需模仿受害模型即可发起攻击。此方法仅需要少量查询来进行训练，在将多任务攻击转化为分类攻击的同时，允许在各种任务间进行有效攻击。 CEMA利用不同的文本分类方法生成多个对抗性候选方案，并选择最能有效攻击替代模型的那一个作为最终输出。实验表明，在涉及两个、三个或六个不同任务（包括分类、翻译、摘要和文本到图像生成）的多任务模型中，即使使用少量的100次查询，CEMA也能实现显著的攻击成功率。 此外，CEMA还能够针对商业API（例如百度和谷歌翻译）、大型语言模型（如ChatGPT 4o）以及图像生成模型（如Stable Diffusion V2）发起有效攻击，展示了其在现实世界应用中的多样性和有效性。
URL
https://arxiv.org/abs/2508.10039
PDF
-
Large Language Models for Oral History Understanding with Text Classification and Sentiment Analysis
2025-08-08 22:06:23Komala Subramanyam Cherukuri, Pranav Abishai Moses, Aisa Sakata, Jiangping Chen, Haihua ChenAbstract
Oral histories are vital records of lived experience, particularly within communities affected by systemic injustice and historical erasure. Effective and efficient analysis of their oral history archives can promote access and understanding of the oral histories. However, Large-scale analysis of these archives remains limited due to their unstructured format, emotional complexity, and high annotation costs. This paper presents a scalable framework to automate semantic and sentiment annotation for Japanese American Incarceration Oral History. Using LLMs, we construct a high-quality dataset, evaluate multiple models, and test prompt engineering strategies in historically sensitive contexts. Our multiphase approach combines expert annotation, prompt design, and LLM evaluation with ChatGPT, Llama, and Qwen. We labeled 558 sentences from 15 narrators for sentiment and semantic classification, then evaluated zero-shot, few-shot, and RAG strategies. For semantic classification, ChatGPT achieved the highest F1 score (88.71%), followed by Llama (84.99%) and Qwen (83.72%). For sentiment analysis, Llama slightly outperformed Qwen (82.66%) and ChatGPT (82.29%), with all models showing comparable results. The best prompt configurations were used to annotate 92,191 sentences from 1,002 interviews in the JAIOH collection. Our findings show that LLMs can effectively perform semantic and sentiment annotation across large oral history collections when guided by well-designed prompts. This study provides a reusable annotation pipeline and practical guidance for applying LLMs in culturally sensitive archival analysis. By bridging archival ethics with scalable NLP techniques, this work lays the groundwork for responsible use of artificial intelligence in digital humanities and preservation of collective memory. GitHub: this https URL.
Abstract (translated)
口述历史是生活经验的重要记录，尤其在那些遭受系统性不公和历史遗忘的社区中。有效且高效地分析这些口述历史档案可以促进对口述史的理解与访问。然而，由于其非结构化格式、情感复杂性和高昂的注释成本，大规模分析仍然受到限制。本文提出了一种可扩展框架，用于自动标注日裔美国人拘留所口述历史的情感和语义。通过使用大语言模型（LLMs），我们构建了一个高质量的数据集，并在敏感的历史背景下评估了多个模型及提示工程策略。 我们的多阶段方法结合了专家注释、设计提示以及与ChatGPT、Llama和Qwen的模型评估。从15位叙述者中，我们将558句话标注为情感分类和语义分类，然后测试了零样本、少样本和RAG（检索增强生成）策略。 对于语义分类，ChatGPT取得了最高的F1分数（88.71%），其次是Llama（84.99%）和Qwen（83.72%）。在情感分析方面，Llama略微优于Qwen（82.66%）和ChatGPT（82.29%），所有模型表现出相似的结果。 我们使用最佳的提示配置注释了日裔美国人拘留所口述历史集合中1,002次采访中的92,191句话。研究结果表明，当通过精心设计的提示引导时，LLMs能够在大规模口述历史收藏上有效地执行语义和情感标注。 这项研究提供了一个可重复使用的注释管道以及在文化敏感档案分析中应用LLMs的实际指导原则。本工作将档案伦理与可扩展自然语言处理技术相结合，为数字人文科学领域中人工智能的负责任使用奠定了基础，并有助于集体记忆的保护。GitHub网址：[此链接](https://github.com/yourusername/repositoryname)（请用实际的GitHub地址替换链接）。
URL
https://arxiv.org/abs/2508.06729
PDF
-
LLMCARE: Alzheimer's Detection via Transformer Models Enhanced by LLM-Generated Synthetic Data
2025-08-08 13:44:55Ali Zolnour, Hossein Azadmaleki, Yasaman Haghbin, Fatemeh Taherinezhad, Mohamad Javad Momeni Nezhad, Sina Rashidi, Masoud Khani, AmirSajjad Taleban, Samin Mahdizadeh Sani, Maryam Dadkhah, James M. Noble, Suzanne Bakken, Yadollah Yaghoobzadeh, Abdol-Hossein Vahabie, Masoud Rouhizadeh, Maryam ZolnooriAbstract
Alzheimer's disease and related dementias (ADRD) affect approximately five million older adults in the U.S., yet over half remain undiagnosed. Speech-based natural language processing (NLP) offers a promising, scalable approach to detect early cognitive decline through linguistic markers. To develop and evaluate a screening pipeline that (i) fuses transformer embeddings with handcrafted linguistic features, (ii) tests data augmentation using synthetic speech generated by large language models (LLMs), and (iii) benchmarks unimodal and multimodal LLM classifiers for ADRD detection. Transcripts from the DementiaBank "cookie-theft" task (n = 237) were used. Ten transformer models were evaluated under three fine-tuning strategies. A fusion model combined embeddings from the top-performing transformer with 110 lexical-derived linguistic features. Five LLMs (LLaMA-8B/70B, MedAlpaca-7B, Ministral-8B, GPT-4o) were fine-tuned to generate label-conditioned synthetic speech, which was used to augment training data. Three multimodal models (GPT-4o, Qwen-Omni, Phi-4) were tested for speech-text classification in zero-shot and fine-tuned settings. The fusion model achieved F1 = 83.3 (AUC = 89.5), outperforming linguistic or transformer-only baselines. Augmenting training data with 2x MedAlpaca-7B synthetic speech increased F1 to 85.7. Fine-tuning significantly improved unimodal LLM classifiers (e.g., MedAlpaca: F1 = 47.3 -> 78.5 F1). Current multimodal models demonstrated lower performance (GPT-4o = 70.2 F1; Qwen = 66.0). Performance gains aligned with the distributional similarity between synthetic and real speech. Integrating transformer embeddings with linguistic features enhances ADRD detection from speech. Clinically tuned LLMs effectively support both classification and data augmentation, while further advancement is needed in multimodal modeling.
Abstract (translated)
阿尔茨海默病及相关痴呆症（ADRD）在美国影响了约500万老年人，然而超过一半的病例仍未被诊断。基于语音的自然语言处理（NLP）为通过语言标志物检测早期认知衰退提供了一种有前景且可扩展的方法。为了开发和评估一种筛查流程，该流程结合了(i) 将变压器嵌入与手工制作的语言特征相融合；(ii) 使用大语言模型（LLM）生成的合成语音进行数据增强测试；以及(iii) 对ADRD检测中的单模态和多模态LLM分类器进行基准测试。使用DementiaBank“偷饼干”任务的转录文本（n = 237）。评估了十种变压器模型，采用了三种不同的微调策略。融合模型结合了表现最佳的变压器的嵌入与110个词汇衍生的语言特征。五个LLM（LLaMA-8B/70B、MedAlpaca-7B、Ministral-8B、GPT-4o）被微调以生成标签条件下的合成语音，该合成语音用于增强训练数据。测试了三种多模态模型（GPT-4o、Qwen-Omni、Phi-4），在零样本和微调设置下进行语音文本分类。融合模型实现了F1 = 83.3（AUC = 89.5）的性能，优于仅使用语言或变压器特征的基本线。用2倍MedAlpaca-7B合成语音增强训练数据后，F1提升至85.7。微调显著提升了单模态LLM分类器的效果（例如，MedAlpaca：F1 = 47.3 -> 78.5 F1）。当前的多模态模型表现出较低的性能（GPT-4o = 70.2 F1；Qwen = 66.0）。性能提升与合成语音和真实语音之间的分布相似性相关。将变压器嵌入与语言特征相结合增强了从语音中检测ADRD的能力。经过临床调优的LLM有效地支持了分类和数据增强，但仍需在多模态建模方面进一步改进。
URL
https://arxiv.org/abs/2508.10027
PDF
-
Hierarchical Text Classification Using Black Box Large Language Models
2025-08-06 08:53:50Kosuke Yoshimura, Hisashi KashimaAbstract
Hierarchical Text Classification (HTC) aims to assign texts to structured label hierarchies; however, it faces challenges due to data scarcity and model complexity. This study explores the feasibility of using black box Large Language Models (LLMs) accessed via APIs for HTC, as an alternative to traditional machine learning methods that require extensive labeled data and computational resources. We evaluate three prompting strategies -- Direct Leaf Label Prediction (DL), Direct Hierarchical Label Prediction (DH), and Top-down Multi-step Hierarchical Label Prediction (TMH) -- in both zero-shot and few-shot settings, comparing the accuracy and cost-effectiveness of these strategies. Experiments on two datasets show that a few-shot setting consistently improves classification accuracy compared to a zero-shot setting. While a traditional machine learning model achieves high accuracy on a dataset with a shallow hierarchy, LLMs, especially DH strategy, tend to outperform the machine learning model on a dataset with a deeper hierarchy. API costs increase significantly due to the higher input tokens required for deeper label hierarchies on DH strategy. These results emphasize the trade-off between accuracy improvement and the computational cost of prompt strategy. These findings highlight the potential of black box LLMs for HTC while underscoring the need to carefully select a prompt strategy to balance performance and cost.
Abstract (translated)
层次文本分类（HTC）旨在将文本分配到结构化的标签层级体系中；然而，由于数据稀缺和模型复杂性的问题，HTC面临着挑战。这项研究探讨了使用通过API访问的黑盒大规模语言模型（LLMs）进行HTC的可行性，作为传统机器学习方法的一种替代方案，后者需要大量的标注数据和计算资源。我们评估了三种提示策略——直接叶标签预测（DL）、直接层级标签预测（DH）以及自顶向下的多步层级标签预测（TMH），在零样本和少量样本设置下进行性能对比，比较这些策略的准确性和成本效益。 实验使用两个数据集表明，在少数样本设置中，分类准确性相比零样本设置始终有所提高。传统机器学习模型在一个浅层层级的数据集中表现出了高精度；然而，LLMs（尤其是DH策略）在深层层级结构的数据集中往往优于传统的机器学习模型。API费用由于更深层次的标签体系所需的输入令牌更多而在DH策略下显著增加。这些结果强调了准确性改进与提示策略计算成本之间的权衡。 这项研究突显了黑盒大规模语言模型在HTC中的潜力，同时也强调了选择合适的提示策略以平衡性能和成本的重要性。
URL
https://arxiv.org/abs/2508.04219
PDF
-
AutoGeTS: Knowledge-based Automated Generation of Text Synthetics for Improving Text Classification
2025-08-04 16:53:20Chenhao Xue, Yuanzhe Jin, Adrian Carrasco-Revilla, Joyraj Chakraborty, Min ChenAbstract
When developing text classification models for real world applications, one major challenge is the difficulty to collect sufficient data for all text classes. In this work, we address this challenge by utilizing large language models (LLMs) to generate synthetic data and using such data to improve the performance of the models without waiting for more real data to be collected and labelled. As an LLM generates different synthetic data in response to different input examples, we formulate an automated workflow, which searches for input examples that lead to more ``effective'' synthetic data for improving the model concerned. We study three search strategies with an extensive set of experiments, and use experiment results to inform an ensemble algorithm that selects a search strategy according to the characteristics of a class. Our further experiments demonstrate that this ensemble approach is more effective than each individual strategy in our automated workflow for improving classification models using LLMs.
Abstract (translated)
在为实际应用开发文本分类模型时，一个主要的挑战是难以收集到所有类别足够的数据。在这项工作中，我们通过利用大型语言模型（LLM）生成合成数据来应对这一挑战，并使用这些数据改进模型性能，而无需等待更多的真实数据被收集和标注。由于LLM根据不同的输入示例会产生不同类型的合成数据，我们制定了一个自动化工作流程，该流程寻找能够产生更有“效果”的合成数据以提升目标模型的输入示例。通过广泛的实验研究了三种搜索策略，并利用实验结果来指导一种集成算法选择适应特定类别的搜索策略。进一步的实验证明，在使用LLM改进分类模型时，这种集成方法比自动化工作流程中的任何单一策略都更有效。
URL
https://arxiv.org/abs/2508.10000
PDF
-
TeSent: A Benchmark Dataset for Fairness-aware Explainable Sentiment Classification in Telugu
2025-08-02 20:42:37Vallabhaneni Raj Kumar, Ashwin S, Supriya Manna, Niladri Sett, Cheedella V S N M S Hema Harshitha, Kurakula Harshitha, Anand Kumar Sharma, Basina Deepakraj, Tanuj Sarkar, Bondada Navaneeth Krishna, Samanthapudi ShakeerAbstract
In the Indian subcontinent, Telugu, one of India's six classical languages, is the most widely spoken Dravidian Language. Despite its 96 million speaker base worldwide, Telugu remains underrepresented in the global NLP and Machine Learning landscape, mainly due to lack of high-quality annotated resources. This work introduces TeSent, a comprehensive benchmark dataset for sentiment classification, a key text classification problem, in Telugu. TeSent not only provides ground truth labels for the sentences, but also supplements with provisions for evaluating explainability and fairness, two critical requirements in modern-day machine learning tasks. We scraped Telugu texts covering multiple domains from various social media platforms, news websites and web-blogs to preprocess and generate 26,150 sentences, and developed a custom-built annotation platform and a carefully crafted annotation protocol for collecting the ground truth labels along with their human-annotated rationales. We then fine-tuned several SOTA pre-trained models in two ways: with rationales, and without rationales. Further, we provide a detailed plausibility and faithfulness evaluation suite, which exploits the rationales, for six widely used post-hoc explainers applied on the trained models. Lastly, we curate TeEEC, Equity Evaluation Corpus in Telugu, a corpus to evaluate fairness of Telugu sentiment and emotion related NLP tasks, and provide a fairness evaluation suite for the trained classifier models. Our experimental results suggest that training with rationales may improve model accuracy, reduce bias in models, and make the explainers' output more aligned to human reasoning.
Abstract (translated)
在印度次大陆，泰卢固语（Telugu）是印度六种古典语言之一，也是使用最广泛的德拉威语系语言。尽管其全球使用者数量达到9600万，但由于高质量标注资源的缺乏，泰卢固语在全球自然语言处理和机器学习领域中仍处于相对边缘化的地位。为此，本研究推出了TeSent，一个全面的基准数据集，用于泰卢固语情感分类这一关键文本分类问题的研究。 TeSent不仅提供了句子的真实标签，还补充了评估可解释性和公平性的机制，这两者是现代机器学习任务中的重要需求。我们从各种社交媒体平台、新闻网站和网络博客中抓取了大量的泰卢固语文本，涵盖多个领域，并预处理生成了26,150个句子。同时，我们建立了一个定制的标注平台以及一个精心设计的标注协议来收集这些真实的标签及其人工标注的理由。 接着，我们在两个方面对几个最先进的预训练模型进行了微调：一是带有理由的情况；二是不带理由的情况。此外，我们还提供了一套详细的合理性与忠实性评估工具，这套工具利用了上述理由，并应用于六种广泛使用的后处理解释器，在训练后的模型上进行应用。 最后，为了评估泰卢固语情感和情绪相关的自然语言处理任务的公平性，我们整理出了TeEEC（Telugu Equity Evaluation Corpus），这是一份专门用于评估此类任务中模型公平性的语料库，并且为这些经过训练的分类器提供了一套公平性评估工具。 实验结果表明，使用理由进行训练可以提高模型的准确性，减少模型中的偏见，并使解释器的输出更加符合人类推理。
URL
https://arxiv.org/abs/2508.01486
PDF
-
Small sample-based adaptive text classification through iterative and contrastive description refinement
2025-08-01 11:12:38Amrit Rajeev, Udayaadithya Avadhanam, Harshula Tulapurkar, SaiBarath SundarAbstract
Zero-shot text classification remains a difficult task in domains with evolving knowledge and ambiguous category boundaries, such as ticketing systems. Large language models (LLMs) often struggle to generalize in these scenarios due to limited topic separability, while few-shot methods are constrained by insufficient data diversity. We propose a classification framework that combines iterative topic refinement, contrastive prompting, and active learning. Starting with a small set of labeled samples, the model generates initial topic labels. Misclassified or ambiguous samples are then used in an iterative contrastive prompting process to refine category distinctions by explicitly teaching the model to differentiate between closely related classes. The framework features a human-in-the-loop component, allowing users to introduce or revise category definitions in natural language. This enables seamless integration of new, unseen categories without retraining, making the system well-suited for real-world, dynamic environments. The evaluations on AGNews and DBpedia demonstrate strong performance: 91% accuracy on AGNews (3 seen, 1 unseen class) and 84% on DBpedia (8 seen, 1 unseen), with minimal accuracy shift after introducing unseen classes (82% and 87%, respectively). The results highlight the effectiveness of prompt-based semantic reasoning for fine-grained classification with limited supervision.
Abstract (translated)
在知识不断演变且类别边界模糊的领域（如票务系统）中，零样本文本分类仍然是一项具有挑战性的任务。大型语言模型（LLM）往往由于话题分离度有限而难以在这种场景下泛化，而少样本方法则受限于数据多样性不足的问题。为此，我们提出了一种结合迭代主题细化、对比提示和主动学习的分类框架。该框架从一组小规模标记样本开始，生成初始的主题标签。之后，通过将误分类或模棱两可的样本用于迭代对比提示过程来进一步优化类别区分度，使模型能够明确地区分出紧密相关的类。 此框架的一个关键特点是包含了一个“人机交互”组件，允许用户以自然语言形式引入或修订类别定义。这使得系统能够在无需重新训练的情况下无缝集成新的、未见过的类别，使其非常适合于现实世界中不断变化的环境。 在AGNews和DBpedia数据集上的评估结果表明了该方法的强大性能：在包含3个已知类和1个未知类的AGNews上达到了91%的准确率，在包含8个已知类和1个未知类的DBpedia上达到了84%的准确率。值得注意的是，引入新的未见过类别后，模型表现基本没有显著下降（分别为82%和87%），这进一步证明了提示语驱动的语义推理在有限监督下的细粒度分类任务中的有效性。
URL
https://arxiv.org/abs/2508.00957
PDF
-
XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML
2025-07-30 10:46:16Ernesto L. Estevanell-Valladares, Suilan Estevez-Velarde, Yoan Guti\'errez, Andr\'es Montoyo, Ruslan MitkovAbstract
Experts in machine learning leverage domain knowledge to navigate decisions in model selection, hyperparameter optimisation, and resource allocation. This is particularly critical for fine-tuning language models (LMs), where repeated trials incur substantial computational overhead and environmental impact. However, no existing automated framework simultaneously tackles the entire model selection and HPO task for resource-efficient LM fine-tuning. We introduce XAutoLM, a meta-learning-augmented AutoML framework that reuses past experiences to optimise discriminative and generative LM fine-tuning pipelines efficiently. XAutoLM learns from stored successes and failures by extracting task- and system-level meta-features to bias its sampling toward fruitful configurations and away from costly dead ends. On four text classification and two question-answering benchmarks, XAutoLM surpasses zero-shot optimiser's peak F1 on five of six tasks, cuts mean evaluation time by up to 4.5x, reduces error ratios by up to sevenfold, and uncovers up to 50% more pipelines above the zero-shot Pareto front. In contrast, simpler memory-based baselines suffer negative transfer. We release XAutoLM and our experience store to catalyse resource-efficient, Green AI fine-tuning in the NLP community.
Abstract (translated)
机器学习专家利用领域知识来指导模型选择、超参数优化和资源配置的决策。特别是在微调语言模型（LMs）时，重复试验会带来巨大的计算开销和环境影响，因此这一过程尤为重要。然而，目前没有现有的自动化框架能够同时处理资源高效的LM微调全过程中的模型选择和超参数优化任务。 我们推出了XAutoLM，这是一个增强型的元学习自动机器学习(AutoML)框架，它通过重用过去的成功与失败经验来高效地优化判别式和生成式语言模型的微调流水线。XAutoLM通过提取任务级和系统级的元特征，从存储的经验中学习，并偏向于采样那些有前途的配置方案，避免无效且成本高昂的努力。 在四个文本分类基准测试和两个问答基准测试上，与零样本优化器相比，XAutoLM在六个任务中的五个任务上实现了最高的F1得分峰值，将平均评估时间缩短了最多4.5倍，减少了错误率高达七倍，并发现多达50%的流水线优于零样本帕累托前沿。 相比之下，较为简单的基于记忆的方法受到负面迁移的影响。我们发布了XAutoLM及其经验存储库，以催化NLP社区中资源高效的绿色AI微调研究和应用。
URL
https://arxiv.org/abs/2508.00924
PDF
-
LLM-based Content Classification Approach for GitHub Repositories by the README Files
2025-07-29 15:09:38Malik Uzair Mehmood, Shahid Hussain, Wen Li Wang, Muhammad Usama MalikAbstract
GitHub is the world's most popular platform for storing, sharing, and managing code. Every GitHub repository has a README file associated with it. The README files should contain project-related information as per the recommendations of GitHub to support the usage and improvement of repositories. However, GitHub repository owners sometimes neglected these recommendations. This prevents a GitHub repository from reaching its full potential. This research posits that the comprehensiveness of a GitHub repository's README file significantly influences its adoption and utilization, with a lack of detail potentially hindering its full potential for widespread engagement and impact within the research community. Large Language Models (LLMs) have shown great performance in many text-based tasks including text classification, text generation, text summarization and text translation. In this study, an approach is developed to fine-tune LLMs for automatically classifying different sections of GitHub README files. Three encoder-only LLMs are utilized, including BERT, DistilBERT and RoBERTa. These pre-trained models are then fine-tuned based on a gold-standard dataset consisting of 4226 README file sections. This approach outperforms current state-of-the-art methods and has achieved an overall F1 score of 0.98. Moreover, we have also investigated the use of Parameter-Efficient Fine-Tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) and shown an economical alternative to full fine-tuning without compromising much performance. The results demonstrate the potential of using LLMs in designing an automatic classifier for categorizing the content of GitHub README files. Consequently, this study contributes to the development of automated tools for GitHub repositories to improve their identifications and potential usages.
Abstract (translated)
GitHub 是全球最受欢迎的代码存储、共享和管理平台。每个 GitHub 仓库都关联有一个 README 文件，根据 GitHub 的推荐应包含项目相关信息以支持仓库的使用与改进。然而，一些 GitHub 仓库的所有者有时会忽视这些推荐，这阻碍了仓库发挥其全部潜力。 本研究认为，GitHub 仓库的 README 文件内容详尽度对其采用和利用具有重要影响。缺乏详细信息可能会限制其在研究社区内的广泛参与和影响力。大型语言模型（LLMs）在许多文本相关任务中表现出色，包括文本分类、生成、摘要以及翻译。在这项研究中，我们开发了一种方法来微调 LLMs 以自动对 GitHub README 文件的不同部分进行分类。所使用的三种编码器模型为 BERT、DistilBERT 和 RoBERTa。然后基于包含 4226 个 README 文件段落的黄金标准数据集对这些预训练模型进行了微调。 这种方法超越了目前最先进的方法，并在总体 F1 得分方面达到了 0.98 的成绩。此外，我们还探讨了使用参数高效微调（PEFT）技术如低秩适应（LoRA），提供了一种经济高效的全量微调替代方案，在不牺牲太多性能的情况下实现目标。结果显示了 LLMs 在设计自动分类器以对 GitHub README 文件内容进行分类的潜力。 因此，这项研究为开发自动化工具改进 GitHub 仓库的识别和潜在使用做出了贡献。
URL
https://arxiv.org/abs/2507.21899
PDF
-
Can human clinical rationales improve the performance and explainability of clinical text classification models?
2025-07-28 19:44:25Christoph Metzner, Shang Gao, Drahomira Herrmannova, Heidi A. HansonAbstract
AI-driven clinical text classification is vital for explainable automated retrieval of population-level health information. This work investigates whether human-based clinical rationales can serve as additional supervision to improve both performance and explainability of transformer-based models that automatically encode clinical documents. We analyzed 99,125 human-based clinical rationales that provide plausible explanations for primary cancer site diagnoses, using them as additional training samples alongside 128,649 electronic pathology reports to evaluate transformer-based models for extracting primary cancer sites. We also investigated sufficiency as a way to measure rationale quality for pre-selecting rationales. Our results showed that clinical rationales as additional training data can improve model performance in high-resource scenarios but produce inconsistent behavior when resources are limited. Using sufficiency as an automatic metric to preselect rationales also leads to inconsistent results. Importantly, models trained on rationales were consistently outperformed by models trained on additional reports instead. This suggests that clinical rationales don't consistently improve model performance and are outperformed by simply using more reports. Therefore, if the goal is optimizing accuracy, annotation efforts should focus on labeling more reports rather than creating rationales. However, if explainability is the priority, training models on rationale-supplemented data may help them better identify rationale-like features. We conclude that using clinical rationales as additional training data results in smaller performance improvements and only slightly better explainability (measured as average token-level rationale coverage) compared to training on additional reports.
Abstract (translated)
基于AI的临床文本分类对于自动检索人群水平健康信息至关重要，并且能够提供可解释性。这项研究探讨了人类生成的临床理由是否可以作为额外监督数据，以提高基于Transformer模型在自动编码临床文档方面性能和可解释性的效果。我们分析了99,125个人类生成的临床理由，这些理由为原发肿瘤部位诊断提供了合理的解释，并将其用作训练样本，与128,649份电子病理报告一起评估用于提取原发癌部位的Transformer模型。此外，我们还研究了充足性作为选择理性质量的一种方式来预先挑选理性的方法。 我们的结果显示，在高资源场景下，将临床理由作为额外的数据进行训练可以提高模型性能，但在资源有限的情况下则会产生不一致的行为。使用充分性作为自动度量标准预选合理性同样会导致结果的不一致性。重要的是，基于合理性的模型始终被额外报告训练出的模型所超越。这表明临床理性并不一贯地提升模型表现，并且在增加更多的报告时会被超越。 因此，如果目标是优化准确率，注解工作应该集中在标记更多报告上而不是创建合理性解释。然而，如果优先考虑的是可解释性，基于补充了理性的数据训练模型可能有助于它们更好地识别类似理由的特征。 综上所述，使用临床理性作为额外的数据进行训练仅在性能改进方面提供较小的优势，并且相对于增加更多报告而言，在提高可解释性（以平均令牌级合理性覆盖率衡量）方面的优势也微乎其微。
URL
https://arxiv.org/abs/2507.21302
PDF
-
Contrast-CAT: Contrasting Activations for Enhanced Interpretability in Transformer-based Text Classifiers
2025-07-27 11:53:01Sungmin Han, Jeonghyun Lee, Sangkyun LeeAbstract
Transformers have profoundly influenced AI research, but explaining their decisions remains challenging -- even for relatively simpler tasks such as classification -- which hinders trust and safe deployment in real-world applications. Although activation-based attribution methods effectively explain transformer-based text classification models, our findings reveal that these methods can be undermined by class-irrelevant features within activations, leading to less reliable interpretations. To address this limitation, we propose Contrast-CAT, a novel activation contrast-based attribution method that refines token-level attributions by filtering out class-irrelevant features. By contrasting the activations of an input sequence with reference activations, Contrast-CAT generates clearer and more faithful attribution maps. Experimental results across various datasets and models confirm that Contrast-CAT consistently outperforms state-of-the-art methods. Notably, under the MoRF setting, it achieves average improvements of x1.30 in AOPC and x2.25 in LOdds over the most competing methods, demonstrating its effectiveness in enhancing interpretability for transformer-based text classification.
Abstract (translated)
Transformer模型对AI研究产生了深远的影响，但解释它们的决策仍然具有挑战性——即使对于相对简单的任务（如分类）也是如此。这阻碍了实际应用中的信任和安全部署。虽然基于激活的方法能够有效解释基于Transformer的文本分类模型，我们的研究表明这些方法可能会受到激活中无关类别的特征干扰，从而导致不那么可靠的解释。为了解决这一限制，我们提出了Contrast-CAT，这是一种新的基于激活对比的归因方法，通过过滤掉无关类别的特征来优化令牌级别的归因。通过将输入序列的激活与参考激活进行比较，Contrast-CAT可以生成更清晰且更准确的归因图谱。 在各种数据集和模型上的实验结果证实了Contrast-CAT始终优于现有的最先进方法。特别地，在MoRF设置下，它分别比最佳竞争方法平均提高了AOPC值1.30倍和LOdds值2.25倍，展示了其增强基于Transformer的文本分类解释性的有效性。
URL
https://arxiv.org/abs/2507.21186
PDF
-
The Role of Orthographic Consistency in Multilingual Embedding Models for Text Classification in Arabic-Script Languages
2025-07-24 19:28:33Abdulhady Abas Abdullah, Amir H. Gandomi, Tarik A Rashid, Seyedali Mirjalili, Laith Abualigah, Milena \v{Z}ivkovi\'c, Hadi VeisiAbstract
In natural language processing, multilingual models like mBERT and XLM-RoBERTa promise broad coverage but often struggle with languages that share a script yet differ in orthographic norms and cultural context. This issue is especially notable in Arabic-script languages such as Kurdish Sorani, Arabic, Persian, and Urdu. We introduce the Arabic Script RoBERTa (AS-RoBERTa) family: four RoBERTa-based models, each pre-trained on a large corpus tailored to its specific language. By focusing pre-training on language-specific script features and statistics, our models capture patterns overlooked by general-purpose models. When fine-tuned on classification tasks, AS-RoBERTa variants outperform mBERT and XLM-RoBERTa by 2 to 5 percentage points. An ablation study confirms that script-focused pre-training is central to these gains. Error analysis using confusion matrices shows how shared script traits and domain-specific content affect performance. Our results highlight the value of script-aware specialization for languages using the Arabic script and support further work on pre-training strategies rooted in script and language specificity.
Abstract (translated)
在自然语言处理领域，像mBERT和XLM-RoBERTa这样的多语种模型承诺广泛覆盖但往往难以应对那些使用相同书写系统却在正字法规范和文化背景上存在差异的语言。这一问题特别突出地表现在阿拉伯字母使用的语言中，如库尔德索拉尼语、阿拉伯语、波斯语和乌尔都语。为此，我们推出了阿拉伯文字RoBERTa（AS-RoBERTa）家族：四个基于RoBERTa的模型，每个模型都是在特定语言的大规模语料库上进行预训练的结果。通过将预训练聚焦于语言特有的书写系统特征和统计信息，我们的模型能够捕捉到通用模型所忽略的语言模式。当针对分类任务进行微调时，AS-RoBERTa变体比mBERT和XLM-RoBERTa的性能高出2至5个百分点。一项消融研究证实了以书写系统为中心的预训练对于这些改进至关重要。利用混淆矩阵进行错误分析揭示了共享书写字母特征及特定领域内容如何影响模型表现。我们的研究成果强调了针对使用阿拉伯字母的语言，具备书写意识的专业化的重要性，并支持进一步探索根植于书写系统和语言特性的预训练策略。
URL
https://arxiv.org/abs/2507.18762
PDF
-
GLiNER2: An Efficient Multi-Task Information Extraction System with Schema-Driven Interface
2025-07-24 16:11:14Urchade Zaratiana, Gil Pasternak, Oliver Boyd, George Hurn-Maloney, Ash LewisAbstract
Information extraction (IE) is fundamental to numerous NLP applications, yet existing solutions often require specialized models for different tasks or rely on computationally expensive large language models. We present GLiNER2, a unified framework that enhances the original GLiNER architecture to support named entity recognition, text classification, and hierarchical structured data extraction within a single efficient model. Built pretrained transformer encoder architecture, GLiNER2 maintains CPU efficiency and compact size while introducing multi-task composition through an intuitive schema-based interface. Our experiments demonstrate competitive performance across extraction and classification tasks with substantial improvements in deployment accessibility compared to LLM-based alternatives. We release GLiNER2 as an open-source pip-installable library with pre-trained models and documentation at this https URL.
Abstract (translated)
信息提取（IE）是众多自然语言处理应用程序的基础，然而现有的解决方案往往需要为不同的任务构建专用模型，或者依赖于计算成本高昂的大规模语言模型。我们提出了GLiNER2，这是一个统一的框架，它增强了最初的GLiNER架构，以便在一个高效的单一模型中支持命名实体识别、文本分类和分层结构化数据提取。基于预先训练的变压器编码器架构，GLiNER2在保持CPU效率和紧凑体积的同时，通过直观的模式基础接口引入了多任务组合功能。我们的实验展示了它在信息提取和分类任务上的竞争力性能，并且与基于大模型的方法相比，在部署便利性方面有了显著改进。我们以开源形式发布了GLiNER2，并将其作为一个可pip安装的库发布，其中包括预训练模型和文档，访问地址为[此处提供的URL]。
URL
https://arxiv.org/abs/2507.18546
PDF
-
Combining Language and Topic Models for Hierarchical Text Classification
2025-07-22 11:45:51Jaco du Toit, Marcel DunaiskiAbstract
Hierarchical text classification (HTC) is a natural language processing task which has the objective of categorising text documents into a set of classes from a predefined structured class hierarchy. Recent HTC approaches use various techniques to incorporate the hierarchical class structure information with the natural language understanding capabilities of pre-trained language models (PLMs) to improve classification performance. Furthermore, using topic models along with PLMs to extract features from text documents has been shown to be an effective approach for multi-label text classification tasks. The rationale behind the combination of these feature extractor models is that the PLM captures the finer-grained contextual and semantic information while the topic model obtains high-level representations which consider the corpus of documents as a whole. In this paper, we use a HTC approach which uses a PLM and a topic model to extract features from text documents which are used to train a classification model. Our objective is to determine whether the combination of the features extracted from the two models is beneficial to HTC performance in general. In our approach, the extracted features are passed through separate convolutional layers whose outputs are combined and passed to a label-wise attention mechanisms which obtains label-specific document representations by weighing the most important features for each class separately. We perform comprehensive experiments on three HTC benchmark datasets and show that using the features extracted from the topic model generally decreases classification performance compared to only using the features obtained by the PLM. In contrast to previous work, this shows that the incorporation of features extracted from topic models for text classification tasks should not be assumed beneficial.
Abstract (translated)
层次文本分类（HTC）是一种自然语言处理任务，其目标是将文本文档归类到预定义的结构化类别层级中的一个或多个类别中。近期的一些HTC方法采用各种技术来整合层次类别结构信息与预先训练的语言模型（PLM）的自然语言理解能力，以提高分类性能。此外，结合使用主题模型和PLM从文本文档中提取特征已被证明是多标签文本分类任务的有效方法。这种组合特征提取模型背后的理由是：PLM能够捕捉到更细致的上下文和语义信息，而主题模型则获取考虑整个文档集合的整体高级表示。 在本文中，我们使用了一种HTC方法，该方法利用一个PLM和一个主题模型从文本文档中提取特征，并用这些特征训练分类模型。我们的目标是确定将这两种模型提取的特征结合起来是否一般而言对HTC性能有益。在我们的方法中，提取到的特征通过各自的卷积层进行处理，然后将其输出组合并通过标签级别的注意力机制获取特定于每个类别的文档表示，该机制根据每种类型的需要为最重要的特征加权。 我们针对三个HTC基准数据集进行了全面实验，并显示使用主题模型提取的特征通常会降低分类性能，而仅仅利用PLM获得的特征则不会。与之前的研究不同的是，这表明在文本分类任务中采用从主题模型提取的特征不应被默认认为是有益的做法。
URL
https://arxiv.org/abs/2507.16490
PDF
-
XL-DURel: Finetuning Sentence Transformers for Ordinal Word-in-Context Classification
2025-07-19 11:40:37Sachin Yadav, Dominik SchlechtwegAbstract
We propose XL-DURel, a finetuned, multilingual Sentence Transformer model optimized for ordinal Word-in-Context classification. We test several loss functions for regression and ranking tasks managing to outperform previous models on ordinal and binary data with a ranking objective based on angular distance in complex space. We further show that binary WiC can be treated as a special case of ordinal WiC and that optimizing models for the general ordinal task improves performance on the more specific binary task. This paves the way for a unified treatment of WiC modeling across different task formulations.
Abstract (translated)
我们提出了XL-DURel，这是一个针对序数词义相似度（Word-in-Context, WiC）分类任务进行了微调的多语言Sentence Transformer模型。我们在回归和排序任务中测试了几种损失函数，并且在处理有序数据和二元数据时，通过基于复数空间中的角度距离进行排名的目标优于之前的模型。此外，我们还表明，二元WiC可以被视为序数WiC的一个特例，并且针对一般序数任务优化的模型可以在更具体的二元任务上提高性能。这为在不同任务设定下统一处理WiC建模铺平了道路。
URL
https://arxiv.org/abs/2507.14578
PDF
-
Evaluating the Effectiveness of Cost-Efficient Large Language Models in Benchmark Biomedical Tasks
2025-07-18 16:13:35Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, Jimmy HuangAbstract
This paper presents a comprehensive evaluation of cost-efficient Large Language Models (LLMs) for diverse biomedical tasks spanning both text and image modalities. We evaluated a range of closed-source and open-source LLMs on tasks such as biomedical text classification and generation, question answering, and multimodal image processing. Our experimental findings indicate that there is no single LLM that can consistently outperform others across all tasks. Instead, different LLMs excel in different tasks. While some closed-source LLMs demonstrate strong performance on specific tasks, their open-source counterparts achieve comparable results (sometimes even better), with additional benefits like faster inference and enhanced privacy. Our experimental results offer valuable insights for selecting models that are optimally suited for specific biomedical applications.
Abstract (translated)
本文对适用于多样化的生物医学任务（涵盖文本和图像模式）的成本效益型大型语言模型（LLMs）进行了全面评估。我们针对生物医学文本分类与生成、问答以及多模态图像处理等任务，测试了一系列的闭源和开源LLMs。我们的实验结果表明，并不存在一个能在所有任务中持续胜出其他模型的单一LLM。相反，不同的LLM在各自的任务上表现出色。尽管某些闭源LLM在特定任务上展示了强大的性能，但其对应的开源版本也能达到类似甚至更好的效果，同时还具备诸如推理速度更快和隐私保护更佳等额外优势。我们的实验结果为选择最适合特定生物医学应用的模型提供了宝贵的见解。
URL
https://arxiv.org/abs/2507.14045
PDF
-
Making Language Model a Hierarchical Classifier and Generator
2025-07-17 09:09:53Yihong Wang, Zhonglin Jiang, Ningyuan Xi, Yue Zhao, Qingqing Gu, Xiyuan Chen, Hao Wu, Sheng Xu, Hange Zhou, Yong Chen, Luo JiAbstract
Decoder-only language models, such as GPT and LLaMA, generally decode on the last layer. Motivated by human's hierarchical thinking capability, we propose that a hierarchical decoder architecture could be built with different layers decoding texts simultaneously. Due to limited time and computationally resources, we choose to adapt a pretrained language model into this form of hierarchical decoder. Language heads of the last layer are copied to different selected intermediate layers, and fine-tuned with different task inputs. By thorough experiments, we validate that these selective intermediate layers could be adapted to speak meaningful and reasonable contents, and this paradigm of hierarchical decoder can obtain state-of-the-art performances on multiple tasks such as hierarchical text classification, classification-guided generation, and hierarchical text generation. This study suggests the possibility of a generalized hierarchical reasoner, pretraining from scratch.
Abstract (translated)
类似于GPT和LLaMA这样的纯解码器语言模型一般在最后一层进行解码。受人类分层思维能力的启发，我们提出可以构建一个分层解码架构，在不同的层次上同时解码文本。由于时间和计算资源有限，我们将选择将预训练的语言模型改编成这种分层解码形式。将最后一层的语言头复制到选定的不同中间层，并使用不同任务输入进行微调。通过全面的实验，我们验证了这些选择性的中间层能够适应并生成有意义且合理的文本内容，并且这种分层解码架构在诸如分层文本分类、指导性生成和分层文本生成等多重任务上可以获得最先进的性能。这项研究表明从零开始预训练一个通用化分层推理模型的可能性。
URL
https://arxiv.org/abs/2507.12930
PDF
-
Post-Training Quantization of Generative and Discriminative LSTM Text Classifiers: A Study of Calibration, Class Balance, and Robustness
2025-07-13 15:48:16Md Mushfiqur Rahaman, Elliot Chang, Tasmiah Haque, Srinjoy DasAbstract
Text classification plays a pivotal role in edge computing applications like industrial monitoring, health diagnostics, and smart assistants, where low latency and high accuracy are both key requirements. Generative classifiers, in particular, have been shown to exhibit robustness to out-of-distribution and noisy data, which is an extremely critical consideration for deployment in such real-time edge environments. However, deploying such models on edge devices faces computational and memory constraints. Post Training Quantization (PTQ) reduces model size and compute costs without retraining, making it ideal for edge deployment. In this work, we present a comprehensive comparative study of generative and discriminative Long Short Term Memory (LSTM)-based text classification models with PTQ using the Brevitas quantization library. We evaluate both types of classifier models across multiple bitwidths and assess their robustness under regular and noisy input conditions. We find that while discriminative classifiers remain robust, generative ones are more sensitive to bitwidth, calibration data used during PTQ, and input noise during quantized inference. We study the influence of class imbalance in calibration data for both types of classifiers, comparing scenarios with evenly and unevenly distributed class samples including their effect on weight adjustments and activation profiles during PTQ. Using test statistics derived from nonparametric hypothesis testing, we identify that using class imbalanced data during calibration introduces insufficient weight adaptation at lower bitwidths for generative LSTM classifiers, thereby leading to degraded performance. This study underscores the role of calibration data in PTQ and when generative classifiers succeed or fail under noise, aiding deployment in edge environments.
Abstract (translated)
文本分类在边缘计算应用中扮演着关键角色，尤其是在工业监控、健康诊断和智能助手等领域，这些领域对低延迟和高准确性都有严格要求。生成式分类器特别展现出对分布外（out-of-distribution）数据及噪声数据的鲁棒性，这是部署于实时边缘环境中的极重要考量因素之一。然而，在边缘设备上部署此类模型会面临计算资源和内存限制的问题。通过使用Post-Training Quantization (PTQ)技术可以在不重新训练的情况下减小模型大小并降低计算成本，这使得该技术非常适合边缘部署。 在本研究中，我们提供了一种基于Brevitas量化库的生成式与判别式长短期记忆（LSTM）文本分类模型的全面比较分析。通过不同位宽(bitwidth)和常规及噪声输入条件对两种类型的分类器进行了评估，并考察了它们的鲁棒性。我们发现尽管判别式分类器依然保持其稳健性，但生成式的分类器则更敏感于位宽、PTQ过程中使用的校准数据以及量化推理期间的输入噪音的影响。 此外，还研究了两类分类器在校准数据中的类别不平衡（class imbalance）影响。通过比较不同类型的样本分布（包括均匀与非均匀分布情况），分析其对权重调整和激活配置文件在PTQ过程中的影响。使用从非参数假设检验得出的统计测试方法发现，在较低位宽时，生成式LSTM分类器在校准过程中使用类别不平衡数据会导致不足的权重重定，从而导致性能下降。 这项研究强调了校准数据在PTQ中的作用以及生成式分类器在噪音环境下的成功或失败情况，这对边缘环境部署具有指导意义。
URL
https://arxiv.org/abs/2507.09687
PDF
-
Enhancing Clinical Text Classification via Fine-Tuned DRAGON Longformer Models
2025-07-13 03:10:19Mingchuan Yang, Ziyuan HuangAbstract
This study explores the optimization of the DRAGON Longformer base model for clinical text classification, specifically targeting the binary classification of medical case descriptions. A dataset of 500 clinical cases containing structured medical observations was used, with 400 cases for training and 100 for validation. Enhancements to the pre-trained joeranbosma/dragon-longformer-base-mixed-domain model included hyperparameter tuning, domain-specific preprocessing, and architectural adjustments. Key modifications involved increasing sequence length from 512 to 1024 tokens, adjusting learning rates from 1e-05 to 5e-06, extending training epochs from 5 to 8, and incorporating specialized medical terminology. The optimized model achieved notable performance gains: accuracy improved from 72.0% to 85.2%, precision from 68.0% to 84.1%, recall from 75.0% to 86.3%, and F1-score from 71.0% to 85.2%. Statistical analysis confirmed the significance of these improvements (p < .001). The model demonstrated enhanced capability in interpreting medical terminology, anatomical measurements, and clinical observations. These findings contribute to domain-specific language model research and offer practical implications for clinical natural language processing applications. The optimized model's strong performance across diverse medical conditions underscores its potential for broad use in healthcare settings.
Abstract (translated)
这项研究探讨了DRAGON Longformer基础模型在临床文本分类中的优化，特别是针对医疗案例描述的二元分类。使用了一个包含500个临床案例的数据集，其中包括结构化的医学观察数据，其中400个案例用于训练，100个案例用于验证。对预训练的joeranbosma/dragon-longformer-base-mixed-domain模型进行了改进，包括超参数调整、领域特定的预处理和架构修改。关键的改动涉及将序列长度从512增加到1024个标记，学习率从1e-05调整到5e-06，训练周期从5延长至8，并且加入了专门的医学术语。 优化后的模型取得了显著的性能提升：准确度从72.0%提高到了85.2%，精度从68.0%提高到了84.1%，召回率从75.0%提高到了86.3%，F1分数从71.0%提高到了85.2%。统计分析确认了这些改进的显著性（p < .001）。该模型在解释医学术语、解剖测量和临床观察方面表现出增强的能力。 这些发现为特定领域的语言模型研究做出了贡献，并对临床自然语言处理应用的实际操作具有重要意义。优化后的模型在各种医疗状况下的优异性能，强调了其在医疗保健环境中广泛使用的潜力。
URL
https://arxiv.org/abs/2507.09470
PDF
