Abstract
Head-mounted displays (HMDs) are essential for experiencing extended reality (XR) environments and observing virtual content. However, they obscure the upper part of the user's face, complicating external video recording and significantly impacting social XR applications such as teleconferencing, where facial expressions and eye gaze details are crucial for creating an immersive experience. This study introduces a geometry-aware learning-based framework to jointly remove HMD occlusions and reconstruct complete 3D facial geometry from RGB frames captured from a single viewpoint. The method integrates a GAN-based video inpainting network, guided by dense facial landmarks and a single occlusion-free reference frame, to restore missing facial regions while preserving identity. Subsequently, a SynergyNet-based module regresses 3D Morphable Model (3DMM) parameters from the inpainted frames, enabling accurate 3D face reconstruction. Dense landmark optimization is incorporated throughout the pipeline to improve both the inpainting quality and the fidelity of the recovered geometry. Experimental results demonstrate that the proposed framework can successfully remove HMDs from RGB facial videos while maintaining facial identity and realism, producing photorealistic 3D face geometry outputs. Ablation studies further show that the framework remains robust across different landmark densities, with only minor quality degradation under sparse landmark configurations.
Abstract (translated)
头戴式显示器（HMD）对于体验扩展现实（XR）环境和观察虚拟内容至关重要。然而，它们遮挡了用户面部的上半部分，这使得外部视频录制变得复杂，并显著影响到诸如远程会议等需要社交互动的XR应用，在这些场景中，面部表情和视线细节对创造沉浸式体验来说是至关重要的因素。 本文介绍了一种基于几何感知的学习框架，该框架能够从单个视角捕获的RGB帧中同时去除HMD遮挡并重建完整的3D面部几何形状。此方法采用了一个由生成对抗网络（GAN）支持的视频修复网络，并通过密集的面部标志和一个无遮挡参考图像来引导缺失区域的恢复过程，以保持身份特征完整。 随后，在修复后的画面基础上，利用基于SynergyNet的方法回归出三维可变形模型（3DMM）参数，从而实现精确的3D人脸重建。在整个流程中融入了密集地标优化步骤，以此提升修复质量和重建几何形状的真实度。 实验结果显示，所提出的框架能够成功地从RGB面部视频中去除HMD遮挡，并保持面部身份和逼真性，产生高度真实的3D脸部几何输出效果。消融研究进一步表明，在不同密度的标记配置下，该框架仍然表现得非常稳健，即使在稀疏地标的情况下也仅有轻微的质量损失。
URL
https://arxiv.org/abs/2508.12336
