Abstract
The recent development of Foundation Models (FMs), represented by large language models, vision transformers, and multimodal models, has been making a significant impact on both academia and industry. Compared with small-scale models, FMs have a much stronger demand for high-volume data during the pre-training phase. Although general FMs can be pre-trained on data collected from open sources such as the Internet, domain-specific FMs need proprietary data, posing a practical challenge regarding the amount of data available due to privacy concerns. Federated Learning (FL) is a collaborative learning paradigm that breaks the barrier of data availability from different participants. Therefore, it provides a promising solution to customize and adapt FMs to a wide range of domain-specific tasks using distributed datasets whilst preserving privacy. This survey paper discusses the potentials and challenges of synergizing FL and FMs and summarizes core techniques, future directions, and applications. A periodically updated paper collection on FM-FL is available at this https URL.
Abstract (translated)
近年来，随着大型语言模型、视觉 transformers 和多模态模型的快速发展，基础模型（FMs）在学术界和产业界都产生了重大影响。与小规模模型相比，FMs 在预训练阶段对大量数据的需求更强。虽然通用 FM 可以预训练在开源数据集上，但领域特定的 FM 需要专有数据，这给隐私问题带来了实际挑战，因为担心数据不足。去中心化学习（FL）是一种合作学习范式，打破了不同参与者数据可用性的障碍。因此，它为通过分布式数据集定制和适应 FM 提供了一个有前途的解决方案，同时保留隐私。本调查论文讨论了 FL 和 FM 的协同作用潜力与挑战，并总结了核心技术、未来方向和应用。目前，关于 FM-FL 的定期更新论文集可在此链接查阅。
URL
https://arxiv.org/abs/2406.12844
