Abstract
We present a new weakly supervised learning-based method for generating novel category-specific 3D shapes from unoccluded image collections. Our method is weakly supervised and only requires silhouette annotations from unoccluded, category-specific objects. Our method does not require access to the object's 3D shape, multiple observations per object from different views, intra-image pixel-correspondences, or any view annotations. Key to our method is a novel multi-projection generative adversarial network (MP-GAN) that trains a 3D shape generator to be consistent with multiple 2D projections of the 3D shapes, and without direct access to these 3D shapes. This is achieved through multiple discriminators that encode the distribution of 2D projections of the 3D shapes seen from a different views. Additionally, to determine the view information for each silhouette image, we also train a view prediction network on visualizations of 3D shapes synthesized by the generator. We iteratively alternate between training the generator and training the view prediction network. We validate our multi-projection GAN on both synthetic and real image datasets. Furthermore, we also show that multi-projection GANs can aid in learning other high-dimensional distributions from lower dimensional training datasets, such as material-class specific spatially varying reflectance properties from images.
Abstract (translated)
我们提出了一种新的基于弱监督学习的方法，用于从未被遮挡的图像集合中生成新的特定于类别的三维形状。我们的方法是弱监控的，只需要来自未包含的、特定于类别的对象的轮廓注释。我们的方法不需要访问对象的三维形状、从不同视图对每个对象进行多次观察、图像内像素对应或任何视图注释。我们的方法的关键是一个新颖的多投影生成对抗网络（MP-GAN），它训练一个三维形状生成器与三维形状的多个二维投影保持一致，并且不直接访问这些三维形状。这是通过对从不同视图看到的三维形状的二维投影分布进行编码的多个鉴别器实现的。此外，为了确定每个轮廓图像的视图信息，我们还训练了一个视图预测网络，该网络用于由生成器合成的三维形状的可视化。我们迭代地交替训练生成器和训练视图预测网络。我们在合成和真实图像数据集上验证了我们的多投影氮化镓。此外，我们还表明，多投影Gas可以帮助从低维训练数据集中学习其他高维分布，例如图像的材料类特定的空间变化反射特性。
URL
https://arxiv.org/abs/1906.03841
