Abstract
Federated learning (FL) has become a promising paradigm for collaborative medical image analysis, yet existing frameworks remain tightly coupled to task-specific backbones and are fragile under heterogeneous imaging modalities. Such constraints hinder real-world deployment, where institutions vary widely in modality distributions and must support diverse downstream tasks. To address this limitation, we propose OmniFM, a modality- and task-agnostic FL framework that unifies training across classification, segmentation, super-resolution, visual question answering, and multimodal fusion without re-engineering the optimization pipeline. OmniFM builds on a key frequency-domain insight: low-frequency spectral components exhibit strong cross-modality consistency and encode modality-invariant anatomical structures. Accordingly, OmniFM integrates (i) Global Spectral Knowledge Retrieval to inject global frequency priors, (ii) Embedding-wise Cross-Attention Fusion to align representations, and (iii) Prefix-Suffix Spectral Prompting to jointly condition global and personalized cues, together regularized by a Spectral-Proximal Alignment objective that stabilizes aggregation. Experiments on real-world datasets show that OmniFM consistently surpasses state-of-the-art FL baselines across intra- and cross-modality heterogeneity, achieving superior results under both fine-tuning and training-from-scratch setups.
Abstract (translated)
联邦学习（FL）已成为协作式医学图像分析的一种有前景范式，但现有框架仍与特定任务架构紧密耦合，且在异构成像模态条件下表现脆弱。此类限制阻碍了真实场景部署——不同机构的模态分布差异显著，且需支持多样化的下游任务。为突破此局限，我们提出OmniFM，一个模态与任务无关的联邦学习框架，可在无需重构优化流程的前提下，统一实现分类、分割、超分辨率、视觉问答及多模态融合等任务的训练。OmniFM基于一个关键频域洞察：低频谱分量呈现强跨模态一致性，并编码模态不变解剖结构。据此，OmniFM整合了（i）全局频谱知识检索以注入全局频率先验，（ii）逐嵌入跨注意力融合以对齐表征，以及（iii）前缀-后缀频谱提示以联合约束全局与个性化线索，并通过频谱近端对齐目标进行正则化以稳定聚合。在真实数据集上的实验表明，OmniFM在模态内与跨模态异构场景下持续超越最先进联邦学习基线，在微调与从头训练设置中均取得更优结果。
URL
https://arxiv.org/abs/2603.21660
