Abstract
We present a deep reinforcement learning method of progressive view inpainting for 3D point scene completion under volume guidance, achieving high-quality scene reconstruction from only a single depth image with severe occlusion. Our approach is end-to-end, consisting of three modules: 3D scene volume reconstruction, 2D depth map inpainting, and multi-view selection for completion. Given a single depth image, our method first goes through the 3D volume branch to obtain a volumetric scene reconstruction as a guide to the next view inpainting step, which attempts to make up the missing information; the third step involves projecting the volume under the same view of the input, concatenating them to complete the current view depth, and integrating all depth into the point cloud. Since the occluded areas are unavailable, we resort to a deep Q-Network to glance around and pick the next best view for large hole completion progressively until a scene is adequately reconstructed while guaranteeing validity. All steps are learned jointly to achieve robust and consistent results. We perform qualitative and quantitative evaluations with extensive experiments on the SUNCG data, obtaining better results than the state of the art.
Abstract (translated)
提出了一种在体导引下完成三维点场景的渐进式视图绘制的深度强化学习方法，实现了单深度图像严重遮挡时的高质量场景重建。我们的方法是端到端的，包括三个模块：三维场景体积重建、二维深度地图绘制和多视图选择完成。对于单一深度的图像，我们的方法首先通过三维体积分支获得一个三维场景重建，作为下一个视图绘制步骤的指导，该步骤试图弥补丢失的信息；第三步涉及到在输入的同一视图下投影体积，将其连接以完成当前视图的绘制。并将所有深度整合到点云中。由于封闭区域不可用，我们借助深度Q网络环顾四周并选择下一个最佳视图，逐步完成大孔，直到场景得到充分重建，同时保证有效性。所有步骤都是共同学习的，以获得稳健和一致的结果。我们对SUNCG数据进行了大量实验，进行了定性和定量评估，获得了比最新技术更好的结果。
URL
https://arxiv.org/abs/1903.04019
