Abstract
Large scale pretrained models have revolutionized Natural Language Processing (NLP) and Computer Vision (CV), showcasing remarkable cross domain generalization abilities. However, in graph learning, models are typically trained on individual graph datasets, limiting their capacity to transfer knowledge across different graphs and tasks. This approach also heavily relies on large volumes of annotated data, which presents a significant challenge in resource-constrained settings. Unlike NLP and CV, graph structured data presents unique challenges due to its inherent heterogeneity, including domain specific feature spaces and structural diversity across various applications. To address these challenges, we propose a novel structure aware self supervised learning method for Text Attributed Graphs (SSTAG). By leveraging text as a unified representation medium for graph learning, SSTAG bridges the gap between the semantic reasoning of Large Language Models (LLMs) and the structural modeling capabilities of Graph Neural Networks (GNNs). Our approach introduces a dual knowledge distillation framework that co-distills both LLMs and GNNs into structure-aware multilayer perceptrons (MLPs), enhancing the scalability of large-scale TAGs. Additionally, we introduce an in-memory mechanism that stores typical graph representations, aligning them with memory anchors in an in-memory repository to integrate invariant knowledge, thereby improving the model's generalization ability. Extensive experiments demonstrate that SSTAG outperforms state-of-the-art models on cross-domain transfer learning tasks, achieves exceptional scalability, and reduces inference costs while maintaining competitive performance.
Abstract (translated)
大规模预训练模型已经在自然语言处理（NLP）和计算机视觉（CV）领域引发了革命，展示了卓越的跨域泛化能力。然而，在图学习中，模型通常在单个图数据集上进行训练，这限制了它们在不同图和任务间迁移知识的能力。这种方法还严重依赖于大量标注数据，在资源受限的情况下带来了重大挑战。与NLP和CV相比，图结构数据由于其固有的异质性（包括特定领域的特征空间及跨各种应用的结构性多样性）而提出了独特的挑战。 为了解决这些难题，我们提出了一种新颖的针对文本属性图（Text Attributed Graphs, TAGs）的感知结构自监督学习方法。通过利用文本作为图学习中的统一表示媒介，SSTAG弥合了大型语言模型（LLMs）的语义推理能力和图神经网络（GNNs）的结构建模能力之间的差距。我们的方法引入了一个双重知识蒸馏框架，将LLMs和GNNs的知识共同蒸馏到感知结构的多层感知机（MLPs）中，从而增强了大规模TAG的可扩展性。此外，我们还提出了一种内存机制，存储典型图表示，并将其与内存仓库中的记忆锚点对齐，以整合不变知识，从而提高模型的泛化能力。 广泛的实验表明，SSTAG在跨域迁移学习任务上超越了当前最先进的方法，在保持竞争性能的同时实现了卓越的可扩展性和推理成本降低。
URL
https://arxiv.org/abs/2510.01248
