Abstract
Surgical scene Multi-Task Federated Learning (MTFL) is essential for robot-assisted minimally invasive surgery (RAS) but remains underexplored in surgical video understanding due to two key challenges: (1) Tissue Diversity: Local models struggle to adapt to site-specific tissue features, limiting their effectiveness in heterogeneous clinical environments and leading to poor local predictions. (2) Task Diversity: Server-side aggregation, relying solely on gradient-based clustering, often produces suboptimal or incorrect parameter updates due to inter-site task heterogeneity, resulting in inaccurate localization. In light of these two issues, we propose SurgFed, a multi-task federated learning framework, enabling federated learning for surgical scene segmentation and depth estimation across diverse surgical types. SurgFed is powered by two appealing designs, i.e., Language-guided Channel Selection (LCS) and Language-guided Hyper Aggregation (LHA), to address the challenge of fully exploration on corss-site and cross-task. Technically, the LCS is first designed a lightweight personalized channel selection network that enhances site-specific adaptation using pre-defined text inputs, which optimally the local model learn the specific embeddings. We further introduce the LHA that employs a layer-wise cross-attention mechanism with pre-defined text inputs to model task interactions across sites and guide a hypernetwork for personalized parameter updates. Extensive empirical evidence shows that SurgFed yields improvements over the state-of-the-art methods in five public datasets across four surgical types. The code is available at this https URL.
Abstract (translated)
手术场景下的多任务联邦学习（MTFL）对于机器人辅助的微创外科手术（RAS）至关重要，但在手术视频理解领域中由于两个关键挑战仍处于未充分探索的状态：(1) 组织多样性：局部模型难以适应特定位置的组织特征，这在异质性临床环境中限制了其效果，并导致预测不佳。(2) 任务多样性：服务器端聚合仅依赖于基于梯度的聚类，在跨站点任务差异的情况下通常会产生次优或错误的参数更新，从而导致定位不准确。鉴于这两个问题，我们提出了SurgFed，这是一种多任务联邦学习框架，使手术场景分割和深度估计在不同类型的手术之间进行分布式联邦学习成为可能。 SurgFed有两个吸引人的设计：语言引导通道选择（LCS）和语言引导超聚合（LHA），以应对跨站点和跨任务充分探索的挑战。技术上来说，首先设计了一个轻量级个性化通道选择网络——LCS，它通过预定义的文本输入增强了特定位置的适应性，并使局部模型能够学习到最合适的嵌入表示。进一步引入了LHA，该机制使用带有预定义文本输入的逐层交叉注意机制来建模跨站点的任务交互并引导超网络进行个性化参数更新。 大量的实验证据表明，在涵盖四种手术类型中的五个公开数据集上，SurgFed相对于现有方法实现了显著改进。代码可在此链接获取：[https URL]（请将[https URL]替换为实际的网址）。
URL
https://arxiv.org/abs/2603.09496
