Abstract
As previous representations for reinforcement learning cannot effectively incorporate a human-intuitive understanding of the 3D environment, they usually suffer from sub-optimal performances. In this paper, we present Semantic-aware Neural Radiance Fields for Reinforcement Learning (SNeRL), which jointly optimizes semantic-aware neural radiance fields (NeRF) with a convolutional encoder to learn 3D-aware neural implicit representation from multi-view images. We introduce 3D semantic and distilled feature fields in parallel to the RGB radiance fields in NeRF to learn semantic and object-centric representation for reinforcement learning. SNeRL outperforms not only previous pixel-based representations but also recent 3D-aware representations both in model-free and model-based reinforcement learning.
Abstract (translated)
以往的强化学习表示无法有效地融入人类直觉理解3D环境,通常表现不如预期。在本文中,我们介绍了语义 aware 神经网络辐射场(SNeRL),它结合卷积编码器,优化了语义 aware 神经网络辐射场(NeRF),以从多视角图像学习3D aware神经网络隐式表示。我们同时介绍了3D语义和蒸馏特征领域,与NeRF的RGB辐射领域并行学习,以在强化学习中学习语义和物体中心表示。SNeRL不仅比过去的像素表示表现更好,在模型无关的强化学习和模型相关的强化学习中也是如此。
URL
https://arxiv.org/abs/2301.11520
