Abstract
Foundation models have revolutionized various fields such as natural language processing (NLP) and computer vision (CV). While efforts have been made to transfer the success of the foundation models in general AI domains to biology, existing works focus on directly adopting the existing foundation model architectures from general machine learning domains without a systematic design considering the unique physicochemical and structural properties of each biological data modality. This leads to suboptimal performance, as these repurposed architectures struggle to capture the long-range dependencies, sparse information, and complex underlying ``grammars'' inherent to biological data. To address this gap, we introduce BioArc, a novel framework designed to move beyond intuition-driven architecture design towards principled, automated architecture discovery for biological foundation models. Leveraging Neural Architecture Search (NAS), BioArc systematically explores a vast architecture design space, evaluating architectures across multiple biological modalities while rigorously analyzing the interplay between architecture, tokenization, and training strategies. This large-scale analysis identifies novel, high-performance architectures, allowing us to distill a set of empirical design principles to guide future model development. Furthermore, to make the best of this set of discovered principled architectures, we propose and compare several architecture prediction methods that effectively and efficiently predict optimal architectures for new biological tasks. Overall, our work provides a foundational resource and a principled methodology to guide the creation of the next generation of task-specific and foundation models for biology.
Abstract (translated)
基础模型已在自然语言处理（NLP）和计算机视觉（CV）等领域引发了革命。尽管人们努力将这些基础模型的成功扩展到生物学领域，但现有的工作主要集中在直接采用通用机器学习领域的现有基础模型架构上，而没有系统地考虑到每种生物数据模态所具有的独特物理化学和结构特性。这种做法导致了次优性能，因为重新调整用途的架构难以捕捉到生物数据中固有的长距离依赖、稀疏信息以及复杂的底层“语法”。 为了解决这一差距，我们引入了一个名为BioArc的新框架，该框架旨在超越直觉驱动的架构设计，转向为生物学基础模型进行原理化和自动化的架构发现。通过利用神经架构搜索（NAS），BioArc系统地探索了庞大的架构设计空间，并在多种生物模态下评估各种架构的同时严谨分析架构、标记化与训练策略之间的相互作用。这种大规模分析识别出新型高效率的架构，使我们能够提炼出一系列实证的设计原则来指导未来模型的发展。 此外，为了最大化利用这一组发现的原则性架构，我们提出了并比较了几种有效的架构预测方法，这些方法可以有效地为新的生物任务预测最优架构。总的来说，我们的工作提供了一个基础资源和一种原理化的方法论，以引导下一代特定任务的和基础生物学模型的创建。
URL
https://arxiv.org/abs/2512.00283
