Abstract
Reinforcement learning (RL) policies deployed in safety-critical systems, such as unmanned aerial vehicle (UAV) navigation in dynamic airspace, are vulnerable to out-ofdistribution (OOD) adversarial attacks in the observation space. These attacks induce distributional shifts that significantly degrade value estimation, leading to unsafe or suboptimal decision making rendering the existing policy fragile. To address this vulnerability, we propose an antifragile RL framework designed to adapt against curriculum of incremental adversarial perturbations. The framework introduces a simulated attacker which incrementally increases the strength of observation-space perturbations which enables the RL agent to adapt and generalize across a wider range of OOD observations and anticipate previously unseen attacks. We begin with a theoretical characterization of fragility, formally defining catastrophic forgetting as a monotonic divergence in value function distributions with increasing perturbation strength. Building on this, we define antifragility as the boundedness of such value shifts and derive adaptation conditions under which forgetting is stabilized. Our method enforces these bounds through iterative expert-guided critic alignment using Wasserstein distance minimization across incrementally perturbed observations. We empirically evaluate the approach in a UAV deconfliction scenario involving dynamic 3D obstacles. Results show that the antifragile policy consistently outperforms standard and robust RL baselines when subjected to both projected gradient descent (PGD) and GPS spoofing attacks, achieving up to 15% higher cumulative reward and over 30% fewer conflict events. These findings demonstrate the practical and theoretical viability of antifragile reinforcement learning for secure and resilient decision-making in environments with evolving threat scenarios.
Abstract (translated)
强化学习（RL）策略在无人驾驶航空器（UAV）导航等安全关键系统中的应用，面对动态空域中观察空间的分布外（OOD）对抗攻击时表现出脆弱性。这些攻击导致价值估计出现显著偏差，从而引发不安全或次优决策，使现有策略变得不稳定。为了应对这一漏洞，我们提出了一种抗脆性的RL框架，该框架能够适应增量式对抗扰动的课程训练。该框架引入了一个模拟攻击者，逐步增加观察空间中的干扰强度，从而使强化学习代理能够在更广泛的OOD观察中进行自适应和泛化，并且可以预测以前未见过的攻击。 我们从理论上对脆弱性进行了描述，正式定义了灾难性遗忘为随着扰动强度增强价值函数分布之间的单调发散。在此基础上，我们将抗脆性定义为这样的价值转移是有限度的，并推导出在哪些适应条件下可以使这种遗忘稳定下来。我们的方法通过使用Wasserstein距离最小化来进行迭代专家指导的批评对齐来强制执行这些限制，该过程是在逐步受到扰动的观察中进行的。 我们在涉及动态3D障碍物规避的UAV场景中进行了实验评估。结果显示，在面对投影梯度下降（PGD）和GPS欺骗攻击时，抗脆性策略在标准RL基准线的基础上累计奖励提高了高达15%，冲突事件减少了超过30%。这些发现证明了抗脆性强化学习对于具有不断变化威胁环境中的安全和弹性决策制定的实用性和理论可行性。 总之，该研究为强化学习算法的安全性和稳健性提供了一种新的方法，并展示了在面对动态且复杂的OOD攻击时，通过适应训练来提高策略鲁棒性的潜力。
URL
https://arxiv.org/abs/2506.21129
