Abstract
Unsupervised pre-training has been on the lookout for the virtue of a value function representation referred to as successor features (SFs), which decouples the dynamics of the environment from the rewards. It has a significant impact on the process of task-specific fine-tuning due to the decomposition. However, existing approaches struggle with local optima due to the unified intrinsic reward of exploration and exploitation without considering the linear regression problem and the discriminator supporting a small skill sapce. We propose a novel unsupervised pre-training model with SFs based on a non-monolithic exploration methodology. Our approach pursues the decomposition of exploitation and exploration of an agent built on SFs, which requires separate agents for the respective purpose. The idea will leverage not only the inherent characteristics of SFs such as a quick adaptation to new tasks but also the exploratory and task-agnostic capabilities. Our suggested model is termed Non-Monolithic unsupervised Pre-training with Successor features (NMPS), which improves the performance of the original monolithic exploration method of pre-training with SFs. NMPS outperforms Active Pre-training with Successor Features (APS) in a comparative experiment.
Abstract (translated)
无监督预训练一直在寻找价值函数表示中被称为后继特征（SFs）的优点，它解耦了环境的动态和奖励。由于分解，它在任务特定微调的过程中具有显著影响。然而，现有方法在局部最优解方面遇到困难，因为没有考虑线性回归问题和支持小技能范围的判别器。我们提出了一种基于非分裂探索方法的非监督预训练模型，其SFs。我们的方法追求在SFs上构建的代理器的探索和利用的分解，这需要分别的代理器。这个想法将不仅利用SF的固有特性，如对新技术快速适应，而且还有探索和任务无关的能力。我们提出的模型称为非分裂无监督预训练 with Successor features (NMPS)，它提高了使用SFs进行预训练的原有聚合物探索方法的性能。在比较实验中，NMPS优于使用成功者特征的主动预训练（APS）。
URL
https://arxiv.org/abs/2405.02569
