Abstract
We propose 360° Volumetric Portrait (3VP) Avatar, a novel method for reconstructing 360° photo-realistic portrait avatars of human subjects solely based on monocular video inputs. State-of-the-art monocular avatar reconstruction methods rely on stable facial performance capturing. However, the common usage of 3DMM-based facial tracking has its limits; side-views can hardly be captured and it fails, especially, for back-views, as required inputs like facial landmarks or human parsing masks are missing. This results in incomplete avatar reconstructions that only cover the frontal hemisphere. In contrast to this, we propose a template-based tracking of the torso, head and facial expressions which allows us to cover the appearance of a human subject from all sides. Thus, given a sequence of a subject that is rotating in front of a single camera, we train a neural volumetric representation based on neural radiance fields. A key challenge to construct this representation is the modeling of appearance changes, especially, in the mouth region (i.e., lips and teeth). We, therefore, propose a deformation-field-based blend basis which allows us to interpolate between different appearance states. We evaluate our approach on captured real-world data and compare against state-of-the-art monocular reconstruction methods. In contrast to those, our method is the first monocular technique that reconstructs an entire 360° avatar.
Abstract (translated)
我们提出了360度立体肖像（3VP）Avatar，这是一种仅基于单目视频输入来重建人类 subject 的 360 度照片现实主义肖像的方法。最先进的单目 Avatar 重建方法依赖于稳定的面部表演捕捉。然而，基于 3DMM 的面部跟踪的常见用法有局限性；侧面视角很难被捕捉到，尤其是在背面视角时，因为缺少面部特征点或人类解析掩码等所需输入。这导致不完整的 Avatar 重建，仅覆盖到前额叶。 相比之下，我们提出了一个基于模板的追踪方案，追踪全身、头部和面部表情，使我们能够从所有侧面覆盖人类 subject 的外观。因此，对于一个在单个相机前旋转的主体的序列，我们基于神经辐射场进行神经体积表示。构建这种表示的一个关键挑战是建模嘴部区域（即嘴唇和牙齿）的外观变化。因此，我们提出了一个变形场为基础的混合基础，使我们能够在不同外观状态之间平滑插值。我们对我们的方法在捕获的现实世界数据上进行评估，并将其与最先进的单目重建方法进行比较。与那些方法相比，我们的方法是第一个仅基于单目的 360 度Avatar 重建方法。
URL
https://arxiv.org/abs/2312.05311
