Abstract
Studies continually find that message-passing graph convolutional networks suffer from the over-smoothing issue. Basically, the issue of over-smoothing refers to the phenomenon that the learned embeddings for all nodes can become very similar to one another and therefore are uninformative after repeatedly applying message passing iterations. Intuitively, we can expect the generated embeddings become smooth asymptotically layerwisely, that is each layer of graph convolution generates a smoothed version of embeddings as compared to that generated by the previous layer. Based on this intuition, we propose RandAlign, a stochastic regularization method for graph convolutional networks. The idea of RandAlign is to randomly align the learned embedding for each node with that of the previous layer using randomly interpolation in each graph convolution layer. Through alignment, the smoothness of the generated embeddings is explicitly reduced. To better maintain the benefit yielded by the graph convolution, in the alignment step we introduce to first scale the embedding of the previous layer to the same norm as the generated embedding and then perform random interpolation for aligning the generated embedding. RandAlign is a parameter-free method and can be directly applied without introducing additional trainable weights or hyper-parameters. We experimentally evaluate RandAlign on different graph domain tasks on seven benchmark datasets. The experimental results show that RandAlign is a general method that improves the generalization performance of various graph convolutional network models and also improves the numerical stability of optimization, advancing the state of the art performance for graph representation learning.
Abstract (translated)
研究表明，消息传递图卷积网络存在过平滑问题。本质上，过平滑是指所有节点的学习嵌入变得非常相似，因此多次应用消息传递迭代后，这些嵌入变得不再具有信息价值。直观上，我们可以预期生成的嵌入在层际上会变得平滑，即与前一层生成的嵌入相比，每个层生成的嵌入都会产生平滑的版本。根据这个直觉，我们提出了RandAlign，一种随机 regularization 方法，用于图卷积网络。RandAlign 的思想是通过在图卷积层中使用随机插值来随机对齐每个节点的学习嵌入，从而减少生成的嵌入的平滑度。为了更好地保持图卷积带来的好处，在对齐步骤中，我们首先将前层的嵌入缩放到与生成的嵌入相同的正则下限，然后对生成的嵌入进行随机插值以进行对齐。RandAlign 是一种无参数方法，可以直接应用而无需引入额外的训练权重或超参数。我们在七个基准数据集上对RandAlign 进行了实验评估。实验结果表明，RandAlign 是一种通用的方法，可以提高各种图卷积网络模型的泛化性能，同时提高优化算法的数值稳定性，推动图形表示学习领域的最新进展。
URL
https://arxiv.org/abs/2404.09774
