Abstract
Foundation models aim to create general, cross-task, and cross-domain machine learning models by pretraining on large-scale datasets to capture shared patterns or concepts (generalities), such as contours, colors, textures, and edges in images, or tokens, words, and sentences in text. However, discovering generalities across graphs remains challenging, which has hindered the development of graph foundation models. To tackle this challenge, in this paper, we propose a novel approach to learn generalities across graphs via task-trees. Specifically, we first define the basic learning instances in graphs as task-trees and assume that the generalities shared across graphs are, at least partially, preserved in the task-trees of the given graphs. To validate the assumption, we first perform a theoretical analysis of task-trees in terms of stability, transferability, and generalization. We find that if a graph neural network (GNN) model is pretrained on diverse task-trees through a reconstruction task, it can learn sufficient transferable knowledge for downstream tasks using an appropriate set of fine-tuning samples. To empirically validate the assumption, we further instantiate the theorems by developing a cross-task, cross-domain graph foundation model named Graph generality Identifier on task-Trees (GIT). The extensive experiments over 30 graphs from five domains demonstrate the effectiveness of GIT in fine-tuning, in-context learning, and zero-shot learning scenarios. Particularly, the general GIT model pretrained on large-scale datasets can be quickly adapted to specific domains, matching or even surpassing expert models designed for those domains. Our data and code are available at this https URL.
Abstract (translated)
基础模型旨在通过在大规模数据集上预训练来创建通用的、跨任务和跨领域的机器学习模型，以捕捉共享模式或概念（普遍性），如图像中的轮廓、颜色、纹理和边缘，或者文本中的标记、单词和句子。然而，在图上发现普遍性仍然是一个挑战，这阻碍了图基础模型的发展。为应对这一挑战，本文提出了一种通过任务树来学习图形间普遍性的新方法。具体来说，我们将图中的基本学习实例定义为任务树，并假设图之间共享的普遍性至少部分保留在给定图的任务树中。为了验证该假设，我们首先从稳定性、可迁移性和泛化能力三个方面对任务树进行了理论分析。发现如果一个图神经网络（GNN）模型通过重建任务在多样化的任务树上进行预训练，它就可以使用一组适当的微调样本学习到足够用于下游任务的可转移知识。为了经验性地验证该假设，我们进一步通过开发一种跨任务、跨领域的图基础模型——基于任务树的图普遍性识别器（GIT），实例化了这些定理。在五个领域中30个图形上的广泛实验表明，GIT在微调、上下文学习和零样本学习场景中的有效性。特别是，预训练于大规模数据集的一般性GIT模型可以快速适应特定领域，并且与专门为那些领域设计的专家模型相比，其表现可匹敌甚至超越后者。我们的数据和代码可以在以下网址获取：[此处提供URL]。
URL
https://arxiv.org/abs/2412.16441
