Abstract
Signal delay poses a fundamental challenge in continuous control and reinforcement learning (RL) by introducing a temporal gap between interaction and perception. Current solutions have largely evolved along two distinct paradigms: model-free approaches which utilize state augmentation to preserve Markovian properties, and model-based methods which focus on inferring latent beliefs via dynamics modeling. In this paper, we bridge these perspectives by introducing State-Action Inpainting Diffuser (SAID), a framework that integrates the inductive bias of dynamics learning with the direct decision-making capability of policy optimization. By formulating the problem as a joint sequence inpainting task, SAID implicitly captures environmental dynamics while directly generating consistent plans, effectively operating at the intersection of model-based and model-free paradigms. Crucially, this generative formulation allows SAID to be seamlessly applied to both online and offline RL. Extensive experiments on delayed continuous control benchmarks demonstrate that SAID achieves state-of-the-art and robust performance. Our study suggests a new methodology to advance the field of RL with delay.
Abstract (translated)
信号延迟在连续控制和强化学习（RL）中构成了一个根本性的挑战，因为它会在交互与感知之间引入时间间隔。目前的解决方案主要沿着两个不同的范式发展：无模型方法通过状态扩充来保持马尔可夫性质；基于模型的方法则侧重于通过动态建模推断潜在信念。在本文中，我们提出了一种新框架——状态-动作填充扩散器（State-Action Inpainting Diffuser, SAID），该框架将动力学学习的归纳偏差与策略优化的直接决策能力相结合，从而弥合了这两种视角之间的差距。通过将其问题表述为联合序列填充任务，SAID可以隐式捕捉环境动态并直接生成一致性的计划，在基于模型和无模型范式的交界处运作。尤为重要的是，这种生成性方法使得SAID能够无缝应用于在线和离线RL中。 在延迟连续控制基准测试中的广泛实验表明，SAID实现了最先进的性能且表现稳定。我们的研究提出了一种新的方法论来推进有延迟情况下的强化学习领域的发展。
URL
https://arxiv.org/abs/2603.01553
