Abstract
Model-based methods have recently shown great potential for off-policy evaluation (OPE); offline trajectories induced by behavioral policies are fitted to transitions of Markov decision processes (MDPs), which are used to rollout simulated trajectories and estimate the performance of policies. Model-based OPE methods face two key challenges. First, as offline trajectories are usually fixed, they tend to cover limited state and action space. Second, the performance of model-based methods can be sensitive to the initialization of their parameters. In this work, we propose the variational latent branching model (VLBM) to learn the transition function of MDPs by formulating the environmental dynamics as a compact latent space, from which the next states and rewards are then sampled. Specifically, VLBM leverages and extends the variational inference framework with the recurrent state alignment (RSA), which is designed to capture as much information underlying the limited training data, by smoothing out the information flow between the variational (encoding) and generative (decoding) part of VLBM. Moreover, we also introduce the branching architecture to improve the model's robustness against randomly initialized model weights. The effectiveness of the VLBM is evaluated on the deep OPE (DOPE) benchmark, from which the training trajectories are designed to result in varied coverage of the state-action space. We show that the VLBM outperforms existing state-of-the-art OPE methods in general.
Abstract (translated)
模型方法最近表现出在非政策评估(OPE)方面的极大潜力。行为政策导致的离线轨迹符合马尔可夫决策过程(MDP)的转移函数,这些轨迹用于生成模拟轨迹并估计政策的性能。基于模型的OPE方法面临两个关键挑战。首先,离线轨迹通常固定,它们往往覆盖有限的状态和行动空间。其次,基于模型的方法的性能可能会 sensitive to theInitialization of their parameters。在本文中,我们提出了 variationallatent branching model(VLBM)来学习MDP的转移函数,通过将环境动态表示为紧凑的latent空间,从其中采样下的状态和奖励。具体来说,VLBM利用和扩展了基于循环状态匹配(RSA)的 variational inference框架,该框架旨在捕捉训练数据限制下尽可能多的信息,通过平滑VLBM中的 variational(编码)和生成(解码)部分之间的信息流。此外,我们还引入了分支架构,以提高模型对随机初始化模型权重的鲁棒性。VLBM的有效性在深度 OPE(DOPE)基准上进行评估,该基准的设计旨在生成不同状态行动空间覆盖面的训练轨迹。我们表明,VLBM普遍优于现有的最先进的 OPE方法。
URL
https://arxiv.org/abs/2301.12056
