Abstract
We propose a control framework that integrates model-based bipedal locomotion with residual reinforcement learning (RL) to achieve robust and adaptive walking in the presence of real-world uncertainties. Our approach leverages a model-based controller, comprising a Divergent Component of Motion (DCM) trajectory planner and a whole-body controller, as a reliable base policy. To address the uncertainties of inaccurate dynamics modeling and sensor noise, we introduce a residual policy trained through RL with domain randomization. Crucially, we employ a model-based oracle policy, which has privileged access to ground-truth dynamics during training, to supervise the residual policy via a novel supervised loss. This supervision enables the policy to efficiently learn corrective behaviors that compensate for unmodeled effects without extensive reward shaping. Our method demonstrates improved robustness and generalization across a range of randomized conditions, offering a scalable solution for sim-to-real transfer in bipedal locomotion.
Abstract (translated)
我们提出了一种控制框架，该框架将基于模型的双足行走与残差强化学习（RL）结合在一起，以实现在现实世界不确定性中的稳健和适应性步行。我们的方法利用了一个基于模型的控制器，包括发散运动成分(DCM)轨迹规划器和全身控制器，作为可靠的基线策略。为了应对动力学建模不准确和传感器噪声等不确定性的挑战，我们引入了一种通过域随机化RL训练得到的残差政策。关键在于，我们使用一个具有访问到真实动力学特权信息的基于模型的预言家策略，在训练期间监督残差策略并通过一种新的监督损失进行指导。这种监督使策略能够高效地学习矫正行为以补偿未建模的影响，而无需大量的奖励塑形。 我们的方法在各种随机条件下表现出增强的鲁棒性和泛化能力，并为双足行走中的仿真到现实转换提供了可扩展解决方案。
URL
https://arxiv.org/abs/2601.16109
