Abstract
Interactive 3D scene generation from a single image has gained significant attention due to its potential to create immersive virtual worlds. However, a key challenge in current 3D generation methods is the limited explorability, which cannot render high-quality images during larger maneuvers beyond the original viewpoint, particularly when attempting to move forward into unseen areas. To address this challenge, we propose WonderFree, the first model that enables users to interactively generate 3D worlds with the freedom to explore from arbitrary angles and directions. Specifically, we decouple this challenge into two key subproblems: novel view quality, which addresses visual artifacts and floating issues in novel views, and cross-view consistency, which ensures spatial consistency across different viewpoints. To enhance rendering quality in novel views, we introduce WorldRestorer, a data-driven video restoration model designed to eliminate floaters and artifacts. In addition, a data collection pipeline is presented to automatically gather training data for WorldRestorer, ensuring it can handle scenes with varying styles needed for 3D scene generation. Furthermore, to improve cross-view consistency, we propose ConsistView, a multi-view joint restoration mechanism that simultaneously restores multiple perspectives while maintaining spatiotemporal coherence. Experimental results demonstrate that WonderFree not only enhances rendering quality across diverse viewpoints but also significantly improves global coherence and consistency. These improvements are confirmed by CLIP-based metrics and a user study showing a 77.20% preference for WonderFree over WonderWorld enabling a seamless and immersive 3D exploration experience. The code, model, and data will be publicly available.
Abstract (translated)
从单张图像生成交互式3D场景因其在创建沉浸式虚拟世界方面的潜力而备受关注。然而，目前的3D生成方法面临的一个关键挑战是探索性有限，在超出原始视点进行更大范围操作（尤其是尝试进入未见区域时）无法渲染高质量图片。为了解决这一难题，我们提出了WonderFree模型，这是首个允许用户以任意角度和方向互动地生成3D世界的模型。具体来说，我们将此挑战分解成两个关键子问题：新颖视角质量和跨视图一致性。前者关注于解决新颖视角中的视觉伪影和漂浮问题；后者则确保不同视角间的空间一致性。 为了提升新视角下的渲染质量，我们引入了WorldRestorer，这是一个基于数据驱动的视频修复模型，旨在消除漂浮物体及其它视觉伪影。此外，我们还提出了一种自动收集训练数据的数据采集管道，为WorldRestorer提供涵盖各种风格场景所需的数据，以适应3D场景生成的需求。 为了提高跨视图的一致性，我们提出了ConsistView，这是一种多视角联合修复机制，在同时恢复多个视角的同时保持时空一致性。实验结果表明，WonderFree不仅在不同视角下提升了渲染质量，而且还显著提高了全局一致性和连贯性。这些改进通过CLIP（一种基于对比学习的预训练模型）基准测试和用户研究得到了验证，结果显示有77.20%的用户更偏好使用WonderFree而非WonderWorld来获得无缝且沉浸式的3D探索体验。 该方法的相关代码、模型及数据将公开发布。
URL
https://arxiv.org/abs/2506.20590
