Abstract
Cooperative Adaptive Cruise Control (CACC) represents a quintessential control strategy for orchestrating vehicular platoon movement within Connected and Automated Vehicle (CAV) systems, significantly enhancing traffic efficiency and reducing energy consumption. In recent years, the data-driven methods, such as reinforcement learning (RL), have been employed to address this task due to their significant advantages in terms of efficiency and flexibility. However, the delay issue, which often arises in real-world CACC systems, is rarely taken into account by current RL-based approaches. To tackle this problem, we propose a Delay-Aware Multi-Agent Reinforcement Learning (DAMARL) framework aimed at achieving safe and stable control for CACC. We model the entire decision-making process using a Multi-Agent Delay-Aware Markov Decision Process (MADA-MDP) and develop a centralized training with decentralized execution (CTDE) MARL framework for distributed control of CACC platoons. An attention mechanism-integrated policy network is introduced to enhance the performance of CAV communication and decision-making. Additionally, a velocity optimization model-based action filter is incorporated to further ensure the stability of the platoon. Experimental results across various delay conditions and platoon sizes demonstrate that our approach consistently outperforms baseline methods in terms of platoon safety, stability and overall performance.
Abstract (translated)
合作自适应巡航控制（CACC）代表了一种在连接和自动驾驶车辆（CAV）系统中协调车辆编队运动的典型控制策略，显著提高了交通效率和降低了能源消耗。近年来，数据驱动的方法，如强化学习（RL），已经被采用来解决这个任务，因为它们在效率和灵活性方面具有显著优势。然而，当前基于RL的方法很少考虑到实世界CACC系统中经常出现的延迟问题。为了解决这个问题，我们提出了一个针对延迟敏感的多代理器强化学习（DAMARL）框架，旨在实现CACC的安全和稳定控制。我们使用多代理器延迟感知马尔可夫决策过程（MADA-MDP）来建模整个决策过程，并开发了一种集中训练和分布式执行（CTDE）的MARL框架，用于分布式控制CACC编队。引入了注意机制的策略网络，以提高CAV通信和决策的性能。此外，还引入了基于速度优化模型的动作滤波器，进一步确保编队的稳定性。在不同的延迟条件和编队大小等实验条件下，我们发现，我们的方法在编队安全、稳定和整体性能方面 consistently超过了基线方法。
URL
https://arxiv.org/abs/2404.15696
