Abstract
Offline reinforcement learning (RL) provides a promising approach to avoid costly online interaction with the real environment. However, the performance of offline RL highly depends on the quality of the datasets, which may cause extrapolation error in the learning process. In many robotic applications, an inaccurate simulator is often available. However, the data directly collected from the inaccurate simulator cannot be directly used in offline RL due to the well-known exploration-exploitation dilemma and the dynamic gap between inaccurate simulation and the real environment. To address these issues, we propose a novel approach to combine the offline dataset and the inaccurate simulation data in a better manner. Specifically, we pre-train a generative adversarial network (GAN) model to fit the state distribution of the offline dataset. Given this, we collect data from the inaccurate simulator starting from the distribution provided by the generator and reweight the simulated data using the discriminator. Our experimental results in the D4RL benchmark and a real-world manipulation task confirm that our method can benefit more from both inaccurate simulator and limited offline datasets to achieve better performance than the state-of-the-art methods.
Abstract (translated)
离线强化学习（RL）提供了一种有前途的方法来避免与真实环境进行昂贵的在线交互。然而，离线RL的性能高度依赖于数据质量，这可能导致学习过程中的扩展误差。在许多机器人应用中，通常缺乏准确的仿真器。然而，由于众所周知的学习-探索困境和仿真器和现实环境之间的动态差距，直接从不准确的仿真器中收集的数据无法直接用于离线RL。为解决这些问题，我们提出了一种结合离线数据和低质量仿真数据的新方法。具体来说，我们预训练了一个生成对抗网络（GAN）模型来适应离线数据的状态分布。然后，我们从生成器提供的分布开始收集离线仿真器的数据，并使用判别器对模拟数据进行重新加权。我们在D4RL基准和现实世界的操作任务中的实验结果证实，我们的方法可以从不准确的仿真器和有限离线数据中获得更好的性能，比最先进的方法具有更高的性能。
URL
https://arxiv.org/abs/2405.04307
