Abstract
In biological tasks, data is rarely plentiful as it is generated from hard-to-gather measurements. Therefore, pre-training foundation models on large quantities of available data and then transfer to low-data downstream tasks is a promising direction. However, how to design effective foundation models for molecular learning remains an open question, with existing approaches typically focusing on models with large parameter capacities. In this work, we propose $\texttt{MiniMol}$, a foundational model for molecular learning with 10 million parameters. $\texttt{MiniMol}$ is pre-trained on a mix of roughly 3300 sparsely defined graph- and node-level tasks of both quantum and biological nature. The pre-training dataset includes approximately 6 million molecules and 500 million labels. To demonstrate the generalizability of $\texttt{MiniMol}$ across tasks, we evaluate it on downstream tasks from the Therapeutic Data Commons (TDC) ADMET group showing significant improvements over the prior state-of-the-art foundation model across 17 tasks. $\texttt{MiniMol}$ will be a public and open-sourced model for future research.
Abstract (translated)
在生物任务中，数据很少是丰富的，因为它们是由难以收集的测量生成的。因此，将大量可用数据上的预训练大型基础模型转移到低数据量的下游任务是一个有前途的方向。然而，如何为分子学习设计有效的基模型仍然是一个未解决的问题，现有的方法通常关注具有大参数容量的模型。在这项工作中，我们提出了 $\texttt{MiniMol}$，一个具有100万个参数的分子学习基础模型。$\texttt{MiniMol}$ 在大约3300个稀疏定义的量子和生物自然水平上进行预训练。预训练数据集包括大约6000万分子和5000万标签。为了证明 $\texttt{MiniMol}$ 在不同任务上的泛化能力，我们在治疗数据共享（TDC）ADMET组中评估了它的下游任务，显示在17个任务上取得了显著的改善，超过了前状态的基模型。$\texttt{MiniMol}$ 将是一个为未来研究公开和开源的模型。
URL
https://arxiv.org/abs/2404.14986
