Abstract
3D panorama synthesis is a promising yet challenging task that demands high-quality and diverse visual appearance and geometry of the generated omnidirectional content. Existing methods leverage rich image priors from pre-trained 2D foundation models to circumvent the scarcity of 3D panoramic data, but the incompatibility between 3D panoramas and 2D single views limits their effectiveness. In this work, we demonstrate that by applying multi-plane synchronization to the operators from 2D foundation models, their capabilities can be seamlessly extended to the omnidirectional domain. Based on this design, we further introduce DreamCube, a multi-plane RGB-D diffusion model for 3D panorama generation, which maximizes the reuse of 2D foundation model priors to achieve diverse appearances and accurate geometry while maintaining multi-view consistency. Extensive experiments demonstrate the effectiveness of our approach in panoramic image generation, panoramic depth estimation, and 3D scene generation.
Abstract (translated)
360度全景图合成是一项既具前景又充满挑战的任务，它要求生成的全方位内容具有高质量且多样的视觉外观和几何结构。现有方法通过利用预训练2D基础模型中的丰富图像先验来克服3D全景数据稀缺的问题，但3D全景与2D单一视图之间的不兼容性限制了这些方法的有效性。在本工作中，我们展示了通过将2D基础模型的操作符应用于多平面同步技术，可以无缝地扩展它们的能力以适用于全方位领域。基于这一设计，我们进一步提出了DreamCube，这是一种用于生成3D全景图的多平面RGB-D扩散模型，它最大化利用了2D基础模型先验知识来实现多样化的外观和准确的几何结构，并同时保持多视角一致性。广泛的实验展示了我们的方法在全景图像生成、全景深度估计以及三维场景生成方面的有效性。
URL
https://arxiv.org/abs/2506.17206
