Abstract
Reinforcement learning algorithms often suffer from poor sample efficiency, making them challenging to apply in multi-task or continual learning settings. Efficiency can be improved by transferring knowledge from a previously trained teacher policy to guide exploration in new but related tasks. However, if the new task sufficiently differs from the teacher's training task, the transferred guidance may be sub-optimal and bias exploration toward low-reward behaviors. We propose an energy-based transfer learning method that uses out-of-distribution detection to selectively issue guidance, enabling the teacher to intervene only in states within its training distribution. We theoretically show that energy scores reflect the teacher's state-visitation density and empirically demonstrate improved sample efficiency and performance across both single-task and multi-task settings.
Abstract (translated)
强化学习算法常常面临样本效率低下的问题，这使得它们在多任务或多阶段学习环境中难以应用。通过从先前训练好的教师策略中转移知识来指导新但相关任务中的探索可以提高效率。然而，如果新的任务与教师训练的任务有显著差异，转移的引导可能会次优，并导致向低收益行为偏移的探索。 我们提出了一种基于能量的迁移学习方法，该方法使用分布外检测（out-of-distribution detection）来选择性地发出指导信号，使教师仅在处于其训练数据分布内的状态下进行干预。理论上，这种能量得分反映了教师访问状态的概率密度，并且实验上表明这种方法能够提升单任务和多任务设置中的样本效率及性能。
URL
https://arxiv.org/abs/2506.16590
