Abstract
Guidance commands of flight vehicles are a series of data sets with fixed time intervals, thus guidance design constitutes a sequential decision problem and satisfies the basic conditions for using deep reinforcement learning (DRL). In this paper, we consider the scenario where the escape flight vehicle (EFV) generates guidance commands based on DRL and the pursuit flight vehicle (PFV) generates guidance commands based on the proportional navigation method. For the EFV, the objective of the guidance design entails progressively maximizing the residual velocity, subject to the constraint imposed by the given evasion distance. Thus an irregular dynamic max-min problem of extremely large-scale is formulated, where the time instant when the optimal solution can be attained is uncertain and the optimum solution depends on all the intermediate guidance commands generated before. For solving this problem, a two-step strategy is conceived. In the first step, we use the proximal policy optimization (PPO) algorithm to generate the guidance commands of the EFV. The results obtained by PPO in the global search space are coarse, despite the fact that the reward function, the neural network parameters and the learning rate are designed elaborately. Therefore, in the second step, we propose to invoke the evolution strategy (ES) based algorithm, which uses the result of PPO as the initial value, to further improve the quality of the solution by searching in the local space. Simulation results demonstrate that the proposed guidance design method based on the PPO algorithm is capable of achieving a residual velocity of 67.24 m/s, higher than the residual velocities achieved by the benchmark soft actor-critic and deep deterministic policy gradient algorithms. Furthermore, the proposed ES-enhanced PPO algorithm outperforms the PPO algorithm by 2.7\%, achieving a residual velocity of 69.04 m/s.
Abstract (translated)
飞行器 guidance 命令是一系列固定时间间隔的数据集，因此 guidance 设计构成了一个序列决策问题，满足了使用深度强化学习（DRL）的基本条件。在本文中，我们考虑了以下场景：逃逸飞行器（EFV）根据 DRL 生成引导命令，而追击飞行器（PFV）根据比例导航方法生成引导命令。对于 EFV，引导设计的目的是在给定逃逸距离的约束下，逐步最大化剩余速度。因此，我们提出了一个极大规模的动态最优最小问题，其中最优解的时间不确定性，而且最优解取决于在之前生成的所有中间引导命令。为了解决这个问题，我们提出了一个两步策略。在第一步，我们使用 Proximal Policy Optimization (PPO) 算法生成 EFV 的引导命令。虽然 PPO 在全局搜索空间中获得的结果粗略，但奖励函数、神经网络参数和学习率都经过了详细设计。因此，在第二步，我们提出了基于 ES 的进化策略（ES）算法，它基于 PPO 的结果作为初始值，在局部空间中进一步改善解决方案的质量。仿真结果表明，基于 PPO 算法的 proposed guidance design method 能够实现 67.24 m/s 的残余速度，高于基准软actor-critic 和 deep deterministic policy gradient 算法获得的残余速度。此外，与 PPO 算法相比，ES-enhanced PPO 算法提高了 2.7%，实现了 69.04 m/s 的残余速度。
URL
https://arxiv.org/abs/2405.03711
