Abstract
The increasing automation of navigation for unmanned aerial vehicles (UAVs) has exposed them to adversarial attacks that exploit vulnerabilities in reinforcement learning (RL) through sensor manipulation. Although existing robust RL methods aim to mitigate such threats, their effectiveness has limited generalization to out-of-distribution shifts from the optimal value distribution, as they are primarily designed to handle fixed perturbation. To address this limitation, this paper introduces an antifragile RL framework that enhances adaptability to broader distributional shifts by incorporating a switching mechanism based on discounted Thompson sampling (DTS). This mechanism dynamically selects among multiple robust policies to minimize adversarially induced state-action-value distribution shifts. The proposed approach first derives a diverse ensemble of action robust policies by accounting for a range of perturbations in the policy space. These policies are then modeled as a multiarmed bandit (MAB) problem, where DTS optimally selects policies in response to nonstationary Bernoulli rewards, effectively adapting to evolving adversarial strategies. Theoretical framework has also been provided where by optimizing the DTS to minimize the overall regrets due to distributional shift, results in effective adaptation against unseen adversarial attacks thus inducing antifragility. Extensive numerical simulations validate the effectiveness of the proposed framework in complex navigation environments with multiple dynamic three-dimensional obstacles and with stronger projected gradient descent (PGD) and spoofing attacks. Compared to conventional robust, non-adaptive RL methods, the antifragile approach achieves superior performance, demonstrating shorter navigation path lengths and a higher rate of conflict-free navigation trajectories compared to existing robust RL techniques
Abstract (translated)
无人驾驶飞机（UAV）导航的自动化程度提高，使其更容易受到通过操纵传感器来利用强化学习（RL）漏洞的对抗性攻击。尽管现有的鲁棒RL方法旨在减轻此类威胁，但由于它们主要设计用于处理固定扰动，因此在面对偏离最优价值分布的变化时，其效果有限且泛化能力不足。为了克服这一限制，本文提出了一种基于折扣Thompson采样（DTS）切换机制的抗脆弱性RL框架，以增强对更广泛分布变化的适应性。此机制通过动态选择多个鲁棒策略来最小化对抗性引起的状态-动作值分布的变化。 所提出的这种方法首先通过对策略空间中一系列扰动进行建模，导出一组多样化的行动鲁棒策略集合。然后将这些策略作为多臂赌博机（MAB）问题进行处理，在此过程中使用DTS根据非平稳的伯努利奖励来优化选择策略，从而有效适应不断变化的对抗性策略。 此外，本文还提供了一个理论框架，通过优化DTS以最小化由于分布变化导致的整体遗憾值，使得该方法能够有效地对抗未见过的对抗性攻击，并因此具有抗脆弱性。大量的数值模拟验证了所提出的框架在复杂导航环境中的有效性，包括存在多个动态三维障碍物的情况，以及更强烈的投影梯度下降（PGD）和欺骗攻击的情况下。 与传统的鲁棒、非适应性的RL方法相比，该抗脆弱性方法展示了更优的性能：其生成的导航路径更短，并且冲突自由的导航轨迹比例更高。这表明，在面对复杂和动态变化的对抗环境中，所提出的框架能够更好地保护UAV的安全性和稳定性。
URL
https://arxiv.org/abs/2506.21127
