Abstract
Text-rich graphs, which exhibit rich textual information on nodes and edges, are prevalent across a wide range of real-world business applications. Large Language Models (LLMs) have demonstrated remarkable abilities in understanding text, which also introduced the potential for more expressive modeling in text-rich graphs. Despite these capabilities, efficiently applying LLMs to representation learning on graphs presents significant challenges. Recently, parameter-efficient fine-tuning methods for LLMs have enabled efficient new task generalization with minimal time and memory consumption. Inspired by this, we introduce Graph-aware Parameter-Efficient Fine-Tuning - GPEFT, a novel approach for efficient graph representation learning with LLMs on text-rich graphs. Specifically, we utilize a graph neural network (GNN) to encode structural information from neighboring nodes into a graph prompt. This prompt is then inserted at the beginning of the text sequence. To improve the quality of graph prompts, we pre-trained the GNN to assist the frozen LLM in predicting the next token in the node text. Compared with existing joint GNN and LMs, our method directly generate the node embeddings from large language models with an affordable fine-tuning cost. We validate our approach through comprehensive experiments conducted on 8 different text-rich graphs, observing an average improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction evaluations. Our results demonstrate the efficacy and efficiency of our model, showing that it can be smoothly integrated with various large language models, including OPT, LLaMA and Falcon.
Abstract (translated)
文本丰富的图在节点和边上表现出丰富的文本信息，这在广泛的现实业务应用中非常普遍。大型语言模型（LLMs）在理解文本方面表现出令人印象深刻的的能力，这也引入了在文本丰富的图中进行更富有表现力的建模的潜力。尽管具有这些能力，将LLM应用于图表示学习仍然存在显著的挑战。最近，参数高效的微调方法为LLM带来了高效的任务泛化，且最小化时间和内存消耗。受到这一启发，我们引入了Graph-aware Parameter-Efficient Fine-Tuning - GPEFT，一种用于在文本丰富的图中以LLM进行高效表示学习的全新方法。具体来说，我们利用图神经网络（GNN）将相邻节点的信息编码成图提示。然后将这个提示插入文本序列的开头。为了提高图提示的质量，我们在预训练GNN的基础上，帮助冻活的LLM预测节点文本中的下一个词。与现有的联合GNN和LM相比，我们的方法可以直接从大型语言模型上以可负担的成本生成节点嵌入。我们在8个不同的文本丰富的图形上进行了全面的实验，观察到平均命中率@1和Mean Reciprocal Rank（MRR）在链路预测评估中的平均提升2%。我们的结果证明了我们的模型的有效性和高效性，表明它可以轻松地与各种大型语言模型，包括OPT，LLLaMA和Falcon集成。
URL
https://arxiv.org/abs/2404.18271
