Abstract
Offline-to-online reinforcement learning (RL), by combining the benefits of offline pretraining and online finetuning, promises enhanced sample efficiency and policy performance. However, existing methods, effective as they are, suffer from suboptimal performance, limited adaptability, and unsatisfactory computational efficiency. We propose a novel framework, PROTO, which overcomes the aforementioned limitations by augmenting the standard RL objective with an iteratively evolving regularization term. Performing a trust-region-style update, PROTO yields stable initial finetuning and optimal final performance by gradually evolving the regularization term to relax the constraint strength. By adjusting only a few lines of code, PROTO can bridge any offline policy pretraining and standard off-policy RL finetuning to form a powerful offline-to-online RL pathway, birthing great adaptability to diverse methods. Simple yet elegant, PROTO imposes minimal additional computation and enables highly efficient online finetuning. Extensive experiments demonstrate that PROTO achieves superior performance over SOTA baselines, offering an adaptable and efficient offline-to-online RL framework.
Abstract (translated)
离线到在线强化学习(RL)，通过结合离线预训练和在线微调的优势，有望提高样本效率和决策表现。然而，现有的方法尽管有效，却面临着最优性能、有限适应性和不满意的计算效率。我们提出了一个 novel 框架，称为 Proto，它能够克服上述限制，通过增加标准RL目标并迭代演化 Regularization term 来增强其性能。采用信任区域更新方式，Proto 能够稳定地初始化微调并实现最佳的最终表现，通过逐渐演化 Regularization term 来放松约束强度。通过仅仅调整几个代码行，Proto 可以桥接任何离线策略预训练和标准非策略RL微调，形成一个强大的离线到在线RL路径，为各种方法带来了极大的适应性。简单但优雅，Proto 仅增加了少量的额外计算，并实现了高效的在线微调。广泛的实验表明，Proto 超越了SOTA基准，提供了一种适应性和高效的离线到在线RL框架。
URL
https://arxiv.org/abs/2305.15669
