Abstract
While Deep Reinforcement Learning (DRL) has emerged as a promising solution for intricate control tasks, the lack of explainability of the learned policies impedes its uptake in safety-critical applications, such as automated driving systems (ADS). Counterfactual (CF) explanations have recently gained prominence for their ability to interpret black-box Deep Learning (DL) models. CF examples are associated with minimal changes in the input, resulting in a complementary output by the DL model. Finding such alternations, particularly for high-dimensional visual inputs, poses significant challenges. Besides, the temporal dependency introduced by the reliance of the DRL agent action on a history of past state observations further complicates the generation of CF examples. To address these challenges, we propose using a saliency map to identify the most influential input pixels across the sequence of past observed states by the agent. Then, we feed this map to a deep generative model, enabling the generation of plausible CFs with constrained modifications centred on the salient regions. We evaluate the effectiveness of our framework in diverse domains, including ADS, Atari Pong, Pacman and space-invaders games, using traditional performance metrics such as validity, proximity and sparsity. Experimental results demonstrate that this framework generates more informative and plausible CFs than the state-of-the-art for a wide range of environments and DRL agents. In order to foster research in this area, we have made our datasets and codes publicly available at this https URL.
Abstract (translated)
尽管深度强化学习（DRL）已成为解决复杂控制任务的的有前景的解决方案，但所学习的策略的可解释性不足，这使得其在关键应用领域（如自动驾驶系统）中的应用受到限制。近年来，由于CF解释器能够解释黑盒深度学习（DL）模型的能力，它们在DL模型的可解释性方面受到了越来越多的关注。CF示例与输入的变化量较小，从而使DL模型具有互补的输出。在解决这种问题之前，尤其是在高维视觉输入的情况下，找到这样的变化是非常具有挑战性的。此外，DL代理商动作对过去状态观察历史依赖所引入的时间依赖性，进一步复杂了CF示例的生成。为了应对这些挑战，我们提出了使用局部重要性图来确定代理在整个过去观察状态序列中的最具影响力输入像素的方法。然后，我们将这个地图输入到深度生成模型中，使得模型的输出在显著区域上进行约束修改，从而生成合理的CF。我们使用传统性能度量标准（如有效性、接近度和稀疏性）评估我们在各种领域的框架的有效性。实验结果表明，我们的框架在广泛的環境和DRL代理商中产生了比现有状态更好的信息量和合理的CF。为了促进该领域的研究，我们将数据集和代码公开发布在https://这个链接上。
URL
https://arxiv.org/abs/2404.18326
