Abstract
High-quality 3D streaming from multiple cameras is crucial for immersive experiences in many AR/VR applications. The limited number of views - often due to real-time constraints - leads to missing information and incomplete surfaces in the rendered images. Existing approaches typically rely on simple heuristics for the hole filling, which can result in inconsistencies or visual artifacts. We propose to complete the missing textures using a novel, application-targeted inpainting method independent of the underlying representation as an image-based post-processing step after the novel view rendering. The method is designed as a standalone module compatible with any calibrated multi-camera system. For this we introduce a multi-view aware, transformer-based network architecture using spatio-temporal embeddings to ensure consistency across frames while preserving fine details. Additionally, our resolution-independent design allows adaptation to different camera setups, while an adaptive patch selection strategy balances inference speed and quality, allowing real-time performance. We evaluate our approach against state-of-the-art inpainting techniques under the same real-time constraints and demonstrate that our model achieves the best trade-off between quality and speed, outperforming competitors in both image and video-based metrics.
Abstract (translated)
高质量的3D流媒体对于许多AR/VR应用中的沉浸式体验至关重要。由于实时限制，来自多个摄像头的有限视图会导致渲染图像中缺失信息和不完整的表面。现有方法通常依赖简单的启发式填充策略来填补这些空白，这可能会导致视觉上的不一致或伪影。我们提出了一种新颖的应用特定的修补方法，用于完成缺失纹理，并将其作为基于图像的后期处理步骤，在新视角渲染后独立于底层表示进行操作。该方法设计为一个与任何校准多摄像头系统兼容的独立模块。 为此，我们引入了一个多视图感知、基于变压器的网络架构，使用时空嵌入以确保帧间的一致性同时保留细节点。此外，我们的分辨率无关设计允许适应不同的摄像机设置，而自适应补丁选择策略则平衡了推断速度和质量，在保证实时性能的同时提供出色的表现。 我们在与最新技术相同的实时限制下评估了我们的方法，并展示了模型在质量和速度之间的最佳权衡，无论是在基于图像还是视频的度量标准上均优于竞争对手。
URL
https://arxiv.org/abs/2603.05507
