Abstract
This paper addresses the problem of visual feature representation learning with an aim to improve the performance of end-to-end reinforcement learning (RL) models. Specifically, a novel architecture is proposed that uses a heterogeneous loss function, called CRC loss, to learn improved visual features which can then be used for policy learning in RL. The CRC-loss function is a combination of three individual loss functions, namely, contrastive, reconstruction and consistency loss. The feature representation is learned in parallel to the policy learning while sharing the weight updates through a Siamese Twin encoder model. This encoder model is augmented with a decoder network and a feature projection network to facilitate computation of the above loss components. Through empirical analysis involving latent feature visualization, an attempt is made to provide an insight into the role played by this loss function in learning new action-dependent features and how they are linked to the complexity of the problems being solved. The proposed architecture, called CRC-RL, is shown to outperform the existing state-of-the-art methods on the challenging Deep mind control suite environments by a significant margin thereby creating a new benchmark in this field.
Abstract (translated)
本文旨在解决视觉特征表示学习的问题,以改善全端强化学习模型的性能。具体而言,我们提出了一种新架构,该架构使用一种称为CRC loss的非均匀损失函数来学习改进的视觉特征,这些特征可以用于全端强化学习中的政策学习。CRC loss函数是三个 individual 损失函数的组合,分别是对比性、重建和一致性损失。特征表示是在政策学习的同时进行的,并通过一个西蒙双编码器模型共享权重更新。该编码器模型被添加了一个解码器和特征投影网络,以方便计算上述损失 components。通过涉及潜在特征可视化的实证分析,我们试图提供 insights into this loss function 在学习新的动作依赖特征中的作用,以及它们如何与解决问题的复杂性相关联。我们提出的架构称为CRC-RL,在挑战性的深度心灵控制套件环境中比现有的先进方法表现更好,从而创造了该领域的新基准。
URL
https://arxiv.org/abs/2301.13473
