Abstract
Real-world reinforcement learning systems must operate under distributional drift in their observation streams, yet most policy architectures implicitly assume fully observed and noise-free states. We study robustness of Proximal Policy Optimization (PPO) under temporally persistent sensor failures that induce partial observability and representation shift. To respond to this drift, we augment PPO with temporal sequence models, including Transformers and State Space Models (SSMs), to enable policies to infer missing information from history and maintain performance. Under a stochastic sensor failure process, we prove a high-probability bound on infinite-horizon reward degradation that quantifies how robustness depends on policy smoothness and failure persistence. Empirically, on MuJoCo continuous-control benchmarks with severe sensor dropout, we show Transformer-based sequence policies substantially outperform MLP, RNN, and SSM baselines in robustness, maintaining high returns even when large fractions of sensors are unavailable. These results demonstrate that temporal sequence reasoning provides a principled and practical mechanism for reliable operation under observation drift caused by sensor unreliability.
Abstract (translated)
现实中的强化学习系统必须在观察流分布漂移的情况下运行，而大多数策略架构则假设完全可观察且无噪声的状态。我们研究了近端策略优化（PPO）在导致部分可观测性和表示迁移的持久传感器故障下的鲁棒性。为了应对这种变化，我们在PPO中加入了时间序列模型，包括Transformer和状态空间模型（SSMs），以使策略能够从历史记录中推断缺失的信息并保持性能。在一个随机的传感器故障过程中，我们证明了无穷时域奖励退化的高概率界，这量化了鲁棒性如何依赖于政策平滑度和失败持久性。在严重传感器丢失的情况下，我们在MuJoCo连续控制基准测试上显示基于Transformer的时间序列策略在鲁棒性方面大幅优于MLP、RNN和SSM基线，在大量传感器不可用时也能保持高回报。这些结果表明，时间序列推理提供了一种原理性和实用性的机制来确保由于传感器不准确性导致的观察漂移下的可靠操作。
URL
https://arxiv.org/abs/2603.04648
