Abstract
The increasing volume of healthcare textual data requires computationally efficient, yet highly accurate classification approaches able to handle the nuanced and complex nature of medical terminology. This research presents Knowledge Distillation for Healthcare Multi-Label Text Classification (KDH-MLTC), a framework leveraging model compression and Large Language Models (LLMs). The proposed approach addresses conventional healthcare Multi-Label Text Classification (MLTC) challenges by integrating knowledge distillation and sequential fine-tuning, subsequently optimized through Particle Swarm Optimization (PSO) for hyperparameter tuning. KDH-MLTC transfers knowledge from a more complex teacher LLM (i.e., BERT) to a lighter student LLM (i.e., DistilBERT) through sequential training adapted to MLTC that preserves the teacher's learned information while significantly reducing computational requirements. As a result, the classification is enabled to be conducted locally, making it suitable for healthcare textual data characterized by sensitivity and, therefore, ensuring HIPAA compliance. The experiments conducted on three medical literature datasets of different sizes, sampled from the Hallmark of Cancer (HoC) dataset, demonstrate that KDH-MLTC achieves superior performance compared to existing approaches, particularly for the largest dataset, reaching an F1 score of 82.70%. Additionally, statistical validation and an ablation study are carried out, proving the robustness of KDH-MLTC. Furthermore, the PSO-based hyperparameter optimization process allowed the identification of optimal configurations. The proposed approach contributes to healthcare text classification research, balancing efficiency requirements in resource-constrained healthcare settings with satisfactory accuracy demands.
Abstract (translated)
不断增加的医疗文本数据量要求计算效率高且准确度高的分类方法，能够处理医学术语的独特复杂性和细微差别。本研究提出了一种利用模型压缩和大型语言模型（LLMs）的知识蒸馏用于医疗多标签文本分类(KDH-MLTC)框架。所提出的这一方法通过整合知识蒸馏与序列微调来解决传统的医疗多标签文本分类(MLTC)挑战，并且使用粒子群优化(PSO)对超参数进行调整以实现进一步的性能优化。KDH-MLTC通过适应于MLTC的序列训练过程，将一个更为复杂的教师模型（如BERT）的知识转移到轻量级的学生模型（例如DistilBERT），这一过程中既保留了教师模型所学的信息又显著降低了计算需求。 因此，在本地进行分类成为可能，这使得该方法非常适合敏感性特征明显的医疗文本数据，并确保符合HIPAA合规要求。研究在不同大小的三个医学文献数据集上进行了实验，这些数据集是从癌症标志性特征（HoC）数据集中抽样得到的。结果表明，KDH-MLTC相比于现有的方法取得了更为优越的表现，尤其是在最大的数据集上达到了82.70%的F1分数。 此外，通过统计验证和消融研究证明了KDH-MLTC的稳健性，并且基于PSO的超参数优化过程允许识别出最佳配置。本研究提出的这一方法为医疗文本分类的研究做出了贡献，在资源受限的医疗环境中平衡计算效率需求的同时满足了对准确性的要求。
URL
https://arxiv.org/abs/2505.07162
