Abstract
The ability to learn and retain a wide variety of tasks is a hallmark of human intelligence that has inspired research in artificial general intelligence. Continual learning approaches provide a significant step towards achieving this goal. It has been known that task variability and context switching are challenging for learning in neural networks. Catastrophic forgetting refers to the poor performance on retention of a previously learned task when a new task is being learned. Switching between different task contexts can be a useful approach to mitigate the same by preventing the interference between the varying task weights of the network. This paper introduces Elastic Weight Generative Networks (EWGN) as an idea for context switching between two different tasks. The proposed EWGN architecture uses an additional network that generates the weights of the primary network dynamically while consolidating the weights learned. The weight generation is input-dependent and thus enables context switching. Using standard computer vision datasets, namely MNIST and fashion-MNIST, we analyse the retention of previously learned task representations in Fully Connected Networks, Convolutional Neural Networks, and EWGN architectures with Stochastic Gradient Descent and Elastic Weight Consolidation learning algorithms. Understanding dynamic weight generation and context-switching ability can be useful in enabling continual learning for improved performance.
Abstract (translated)
学习和保留广泛任务的能力是人类智能的一个标志，这激发了对通用人工智能的研究。连续学习的方法为实现这一目标提供了重要的步骤。已知任务多样性和上下文切换对于神经网络中的学习构成了挑战。“灾难性遗忘”指的是当学习新任务时，以前所学任务的表现会变得糟糕的现象。在不同任务的背景下进行切换可以通过防止网络中变化的任务权重之间的干扰来减轻这种问题。 本文介绍了一种名为弹性权重生成网络（Elastic Weight Generative Networks, EWGN）的方法，用于在两个不同的任务之间实现上下文切换。提出的EWGN架构使用一个附加网络动态地生成主网络的权重，并同时巩固已经学习到的权重。该权重生成是基于输入依赖的，因此可以实现上下文切换。 我们利用标准计算机视觉数据集（如MNIST和时尚-MNIST），分析在全连接网络、卷积神经网络以及采用随机梯度下降法和弹性权重巩固算法的EWGN架构中之前学习到的任务表示的保留情况。理解动态权重生成和上下文转换能力对于实现连续学习，提高性能很有帮助。
URL
https://arxiv.org/abs/2506.02065
