Abstract
Federated Learning (FL) faces major challenges regarding communication overhead and model privacy when training large language models (LLMs), especially in healthcare applications. To address these, we introduce Selective Attention Federated Learning (SAFL), a novel approach that dynamically fine-tunes only those transformer layers identified as attention-critical. By employing attention patterns to determine layer importance, SAFL significantly reduces communication bandwidth and enhances differential privacy resilience. Evaluations on clinical NLP benchmarks (i2b2 Clinical Concept Extraction and MIMIC-III discharge summaries) demonstrate that SAFL achieves competitive performance with centralized models while substantially improving communication efficiency and privacy preservation.
Abstract (translated)
联邦学习（FL）在训练大型语言模型（LLMs）时，特别是在医疗应用中，面临着通信开销和模型隐私的重大挑战。为了解决这些问题，我们引入了一种新颖的方法——选择性注意力联邦学习（SAFL），该方法动态地只对那些被识别为关键注意层的变换器层进行微调。通过利用注意力模式来确定各层的重要性，SAFL显著减少了通信带宽，并增强了差分隐私的抵抗能力。在临床自然语言处理基准测试（i2b2临床概念抽取和MIMIC-III出院总结）上的评估表明，SAFL能够在保持与集中式模型相当性能的同时，大幅提高通信效率并保护隐私。
URL
https://arxiv.org/abs/2504.11793
