Abstract
Deep Reinforcement Learning (RL) has demonstrated success in solving complex sequential decision-making problems by integrating neural networks with the RL framework. However, training deep RL models poses several challenges, such as the need for extensive hyperparameter tuning and high computational costs. Transfer learning has emerged as a promising strategy to address these challenges by enabling the reuse of knowledge from previously learned tasks for new, related tasks. This avoids the need for retraining models entirely from scratch. A commonly used approach for transfer learning in RL is to leverage the internal representations learned by the neural network during training. Specifically, the activations from the last hidden layer can be viewed as refined state representations that encapsulate the essential features of the input. In this work, we investigate whether these representations can be used as input for training simpler models, such as linear function approximators, on new tasks. We observe that the representations learned by standard deep RL models can be highly correlated, which limits their effectiveness when used with linear function approximation. To mitigate this problem, we propose a novel deep Q-learning approach that introduces a regularization term to reduce positive correlations between feature representation of states. By leveraging these reduced correlated features, we enable more effective use of linear function approximation in transfer learning. Through experiments and ablation studies on standard RL benchmarks and MinAtar games, we demonstrate the efficacy of our approach in improving transfer learning performance and thereby reducing computational overhead.
Abstract (translated)
深度强化学习（RL）通过将神经网络与RL框架相结合，成功解决了复杂的序列决策问题。然而，训练深度RL模型存在一些挑战，例如需要进行广泛的超参数调整和高昂的计算成本。迁移学习作为一种有前景的方法已经出现，它能够使模型在执行新任务时复用之前任务中学到的知识，从而避免完全从头开始重新训练模型的需求。 在RL中的一个常用的迁移学习方法是利用神经网络在训练过程中学到的内部表示。具体来说，最后一层隐藏层的激活可以被视为细化的状态表示，这些状态表示封装了输入的关键特征。在这项工作中，我们探讨了是否可以通过使用这些表示作为新任务中较简单模型（如线性函数近似器）的输入来进行学习。 我们的研究发现，标准深度RL模型学到的表示可能具有高度的相关性，这在与线性函数近似的结合时会限制其有效性。为了解决这个问题，我们提出了一种新的深度Q学习方法，该方法引入了一个正则化项来减少状态特征表示之间的正相关关系。通过利用这些较低相关的特征，我们能够更有效地使用线性函数近似来进行迁移学习。 我们在标准RL基准和MinAtar游戏中进行了实验和消融研究，结果表明我们的方法在提高迁移学习性能方面是有效的，并且因此降低了计算开销。
URL
https://arxiv.org/abs/2509.24947
