Abstract
The advancements in artificial intelligence in recent years, such as Large Language Models (LLMs), have fueled expectations for breakthroughs in genomic foundation models (GFMs). The code of nature, hidden in diverse genomes since the very beginning of life's evolution, holds immense potential for impacting humans and ecosystems through genome modeling. Recent breakthroughs in GFMs, such as Evo, have attracted significant investment and attention to genomic modeling, as they address long-standing challenges and transform in-silico genomic studies into automated, reliable, and efficient paradigms. In the context of this flourishing era of consecutive technological revolutions in genomics, GFM studies face two major challenges: the lack of GFM benchmarking tools and the absence of open-source software for diverse genomics. These challenges hinder the rapid evolution of GFMs and their wide application in tasks such as understanding and synthesizing genomes, problems that have persisted for decades. To address these challenges, we introduce GFMBench, a framework dedicated to GFM-oriented benchmarking. GFMBench standardizes benchmark suites and automates benchmarking for a wide range of open-source GFMs. It integrates millions of genomic sequences across hundreds of genomic tasks from four large-scale benchmarks, democratizing GFMs for a wide range of in-silico genomic applications. Additionally, GFMBench is released as open-source software, offering user-friendly interfaces and diverse tutorials, applicable for AutoBench and complex tasks like RNA design and structure prediction. To facilitate further advancements in genome modeling, we have launched a public leaderboard showcasing the benchmark performance derived from AutoBench. GFMBench represents a step toward standardizing GFM benchmarking and democratizing GFM applications.
Abstract (translated)
近年来人工智能在基因领域的进步，如大型语言模型（LLMs），已经推动了基因组基础模型（GFMs）突破的期望。藏于生命进化之初的基因代码，具有通过基因组建模对人类和生态系统产生巨大影响潜力。最近GFM的突破，如Evo，吸引了大量投资和关注基因建模，因为它们解决了长期挑战，将仿真基因组研究转化为自动、可靠、高效的范式。在当前不断繁荣的基因组技术革命背景下，GFM研究面临两个主要挑战：缺乏GFM基准测试工具和缺乏多样性的基因组软件。这些挑战阻碍了GFM的快速发展和其在诸如理解和合成基因组等任务上的广泛应用，这些问题已经持续了几十年。为了应对这些挑战，我们介绍了GFMBench，一个专注于GFM导向基准测试的框架。GFMBench标准化基准集并自动为多种开源GFM进行基准测试。它从四个大型基准中整合了数百万个基因组序列，为广泛的基因组应用（如GFM）提供了民主化的GFM。此外，GFMBench作为开源软件发布，提供了用户友好的界面和丰富的教程，适用于AutoBench和复杂的任务（如RNA设计结构预测）。为了推动进一步的基因组建模，我们已经推出了基于AutoBench的公开领导者板，展示基准性能。GFMBench标志着将GFM基准测试标准化和民主化的工作向前迈进了一步。
URL
https://arxiv.org/abs/2410.01784
