Scene_Generation
-
DreamAnywhere: Object-Centric Panoramic 3D Scene Generation
2025-06-25 12:30:41Edoardo Alberto Dominici, Jozef Hladky, Floor Verhoeven, Lukas Radl, Thomas Deixelberger, Stefan Ainetter, Philipp Drescher, Stefan Hauswiesner, Arno Coomans, Giacomo Nazzaro, Konstantinos Vardis, Markus SteinbergerAbstract
Recent advances in text-to-3D scene generation have demonstrated significant potential to transform content creation across multiple industries. Although the research community has made impressive progress in addressing the challenges of this complex task, existing methods often generate environments that are only front-facing, lack visual fidelity, exhibit limited scene understanding, and are typically fine-tuned for either indoor or outdoor settings. In this work, we address these issues and propose DreamAnywhere, a modular system for the fast generation and prototyping of 3D scenes. Our system synthesizes a 360° panoramic image from text, decomposes it into background and objects, constructs a complete 3D representation through hybrid inpainting, and lifts object masks to detailed 3D objects that are placed in the virtual environment. DreamAnywhere supports immersive navigation and intuitive object-level editing, making it ideal for scene exploration, visual mock-ups, and rapid prototyping -- all with minimal manual modeling. These features make our system particularly suitable for low-budget movie production, enabling quick iteration on scene layout and visual tone without the overhead of traditional 3D workflows. Our modular pipeline is highly customizable as it allows components to be replaced independently. Compared to current state-of-the-art text and image-based 3D scene generation approaches, DreamAnywhere shows significant improvements in coherence in novel view synthesis and achieves competitive image quality, demonstrating its effectiveness across diverse and challenging scenarios. A comprehensive user study demonstrates a clear preference for our method over existing approaches, validating both its technical robustness and practical usefulness.
Abstract (translated)
近期在文本到3D场景生成领域的进展展示了其有潜力彻底改变多个行业的内容创作。尽管研究社区在这项复杂任务上取得了令人瞩目的进步，但现有方法通常只能生成面向前方的环境，并且缺乏视觉真实感、场景理解能力有限，而且通常是为室内或室外环境进行微调的。在本工作中，我们解决了这些问题并提出了DreamAnywhere系统——这是一个用于快速生成和原型设计3D场景的模块化系统。 我们的系统从文本中合成一个360°全景图像，将其分解成背景和物体，并通过混合填充技术构建完整的3D表示。此外，它还能够将对象掩模提升为详细且放置在虚拟环境中的3D对象。DreamAnywhere支持沉浸式导航和直观的对象级编辑功能，非常适合场景探索、视觉模拟以及快速原型设计——所有这些都可以用最少的手动建模完成。 该系统的这些特点特别适合低预算电影制作，因为它允许创作者迅速迭代场景布局和视觉风格，而无需传统3D工作流程带来的额外负担。我们的模块化管道高度可定制，因为它的组件可以独立替换。与目前最先进的基于文本和图像的3D场景生成方法相比，DreamAnywhere在新视角合成的一致性和图像质量方面显示出显著改进，证明了它在各种具有挑战性的场景中的有效性。 一项全面的用户研究显示，相对于现有方法，用户更倾向于使用我们的方法，这验证了其技术稳定性和实际应用价值。
URL
https://arxiv.org/abs/2506.20367
PDF
-
From 2D to 3D Cognition: A Brief Survey of General World Models
2025-06-25 05:05:09Ningwei Xie, Zizi Tian, Lei Yang, Xiao-Ping Zhang, Meng Guo, Jie LiAbstract
World models have garnered increasing attention in the development of artificial general intelligence (AGI), serving as computational frameworks for learning representations of the external world and forecasting future states. While early efforts focused on 2D visual perception and simulation, recent 3D-aware generative world models have demonstrated the ability to synthesize geometrically consistent, interactive 3D environments, marking a shift toward 3D spatial cognition. Despite rapid progress, the field lacks systematic analysis to categorize emerging techniques and clarify their roles in advancing 3D cognitive world models. This survey addresses this need by introducing a conceptual framework, providing a structured and forward-looking review of world models transitioning from 2D perception to 3D cognition. Within this framework, we highlight two key technological drivers, particularly advances in 3D representations and the incorporation of world knowledge, as fundamental pillars. Building on these, we dissect three core cognitive capabilities that underpin 3D world modeling: 3D physical scene generation, 3D spatial reasoning, and 3D spatial interaction. We further examine the deployment of these capabilities in real-world applications, including embodied AI, autonomous driving, digital twin, and gaming/VR. Finally, we identify challenges across data, modeling, and deployment, and outline future directions for advancing more robust and generalizable 3D world models.
Abstract (translated)
世界模型在人工通用智能（AGI）的发展中获得了越来越多的关注，它们作为计算框架用于学习外部世界的表示并预测未来的状态。早期的努力主要集中在二维视觉感知和模拟上，而最近基于三维感知的生成型世界模型则展示了合成几何一致且可互动的3D环境的能力，标志着向三维空间认知的重大转变。尽管这一领域取得了迅速的进步，但缺乏系统性的分析来分类新兴技术，并明确它们在推进三维认知世界模型中的作用。本综述旨在通过引入概念框架，提供一个结构化和前瞻性的关于从二维感知到三维认知的世界模型的审查，以满足这一需求。 在这个框架内，我们强调了两项关键的技术驱动力：三维表示的进步和世界知识的整合，作为基础支柱。在此基础上，我们将剖析三个核心的认知能力，这些能力支撑着3D世界的建模：3D物理场景生成、3D空间推理以及3D空间互动。此外，我们将进一步探讨在现实应用中部署这些能力的情况，包括具身人工智能、自动驾驶汽车、数字孪生及游戏/虚拟现实等。 最后，我们识别了跨数据、模型和部署层面的挑战，并概述了推进更强大且通用化的三维世界模型的未来方向。
URL
https://arxiv.org/abs/2506.20134
PDF
-
CoCo4D: Comprehensive and Complex 4D Scene Generation
2025-06-24 17:05:44Junwei Zhou, Xueting Li, Lu Qi, Ming-Hsuan YangAbstract
Existing 4D synthesis methods primarily focus on object-level generation or dynamic scene synthesis with limited novel views, restricting their ability to generate multi-view consistent and immersive dynamic 4D scenes. To address these constraints, we propose a framework (dubbed as CoCo4D) for generating detailed dynamic 4D scenes from text prompts, with the option to include images. Our method leverages the crucial observation that articulated motion typically characterizes foreground objects, whereas background alterations are less pronounced. Consequently, CoCo4D divides 4D scene synthesis into two responsibilities: modeling the dynamic foreground and creating the evolving background, both directed by a reference motion sequence. Given a text prompt and an optional reference image, CoCo4D first generates an initial motion sequence utilizing video diffusion models. This motion sequence then guides the synthesis of both the dynamic foreground object and the background using a novel progressive outpainting scheme. To ensure seamless integration of the moving foreground object within the dynamic background, CoCo4D optimizes a parametric trajectory for the foreground, resulting in realistic and coherent blending. Extensive experiments show that CoCo4D achieves comparable or superior performance in 4D scene generation compared to existing methods, demonstrating its effectiveness and efficiency. More results are presented on our website this https URL.
Abstract (translated)
现有的4D合成方法主要集中在对象级别的生成或具有有限新颖视角的动态场景合成上，这限制了它们生成多视角一致和沉浸式的动态4D场景的能力。为了解决这些局限性，我们提出了一种框架（称为CoCo4D），该框架可以根据文本提示生成详细的动态4D场景，并可选择性地包含图像。我们的方法基于一个关键观察：连贯运动通常表征前景对象的特性，而背景的变化则相对较少。因此，CoCo4D将4D场景合成分为两个任务：建模动态前景和创造不断变化的背景，两者都由参考运动序列指导。 给定文本提示及可选参考图像后，CoCo4D首先使用视频扩散模型生成初始运动序列。接着，该运动序列指导动态前景对象和背景的合成过程，采用了一种新颖的渐进式扩展方案（progressive outpainting scheme）。为了确保移动前景物体在动态背景下无缝融合，CoCo4D优化了前景路径的参数化轨迹，从而实现了现实感强且连贯的合成效果。 广泛的实验表明，与现有方法相比，CoCo4D在生成4D场景方面取得了相当或更好的性能表现，证明了其有效性和效率。更多结果可在我们的网站上查看：[此URL]（请将"this https URL"替换为实际链接）。
URL
https://arxiv.org/abs/2506.19798
PDF
-
PrITTI: Primitive-based Generation of Controllable and Editable 3D Semantic Scenes
2025-06-23 20:47:18Christina Ourania Tze, Daniel Dauner, Yiyi Liao, Dzmitry Tsishkou, Andreas GeigerAbstract
Large-scale 3D semantic scene generation has predominantly relied on voxel-based representations, which are memory-intensive, bound by fixed resolutions, and challenging to edit. In contrast, primitives represent semantic entities using compact, coarse 3D structures that are easy to manipulate and compose, making them an ideal representation for this task. In this paper, we introduce PrITTI, a latent diffusion-based framework that leverages primitives as the main foundational elements for generating compositional, controllable, and editable 3D semantic scene layouts. Our method adopts a hybrid representation, modeling ground surfaces in a rasterized format while encoding objects as vectorized 3D primitives. This decomposition is also reflected in a structured latent representation that enables flexible scene manipulation of ground and object components. To overcome the orientation ambiguities in conventional encoding methods, we introduce a stable Cholesky-based parameterization that jointly encodes object size and orientation. Experiments on the KITTI-360 dataset show that PrITTI outperforms a voxel-based baseline in generation quality, while reducing memory requirements by up to $3\times$. In addition, PrITTI enables direct instance-level manipulation of objects in the scene and supports a range of downstream applications, including scene inpainting, outpainting, and photo-realistic street-view synthesis.
Abstract (translated)
大规模的三维语义场景生成主要依赖于基于体素（voxel）的表示方法，这种方法内存消耗大、分辨率固定且难以编辑。相比之下，原语（primitives）使用紧凑而粗糙的3D结构来表示语义实体，易于操作和组合，成为这一任务的理想选择。在本文中，我们介绍了一种名为PrITTI的新框架，该框架采用基于潜在扩散的方法，并以原语作为生成组成化、可控性和可编辑性三维语义场景布局的主要基础元素。我们的方法采用了混合表示方式，在栅格格式下建模地面表面的同时将物体编码为向量化的3D原语。这种分解也反映在一种结构化的潜在表示中，该表示能够灵活地操作场景中的地面和对象组件。 为了克服传统编码方法中存在的方位模糊性问题，我们引入了一种基于稳定的乔累斯基（Cholesky）参数化的方法，这种方法同时编码物体的大小和方向。实验结果表明，在KITTI-360数据集上的生成质量方面，PrITTI优于基于体素的基础线模型，并且将内存需求减少了最多三倍。此外，PrITTI支持场景中对象实例级别的直接操作，并适用于包括场景修复、扩展以及合成照片级真实街道视图在内的多种下游应用。
URL
https://arxiv.org/abs/2506.19117
PDF
-
VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory
2025-06-23 17:59:56Runjia Li, Philip Torr, Andrea Vedaldi, Tomas JakabAbstract
We propose a novel memory mechanism to build video generators that can explore environments interactively. Similar results have previously been achieved by out-painting 2D views of the scene while incrementally reconstructing its 3D geometry, which quickly accumulates errors, or by video generators with a short context window, which struggle to maintain scene coherence over the long term. To address these limitations, we introduce Surfel-Indexed View Memory (VMem), a mechanism that remembers past views by indexing them geometrically based on the 3D surface elements (surfels) they have observed. VMem enables the efficient retrieval of the most relevant past views when generating new ones. By focusing only on these relevant views, our method produces consistent explorations of imagined environments at a fraction of the computational cost of using all past views as context. We evaluate our approach on challenging long-term scene synthesis benchmarks and demonstrate superior performance compared to existing methods in maintaining scene coherence and camera control.
Abstract (translated)
我们提出了一种新颖的记忆机制，用于构建能够进行交互式环境探索的视频生成器。此前曾通过扩展场景的二维视图并逐步重建其三维几何结构来实现类似结果，但这种方法会迅速积累误差，或者使用短上下文窗口的视频生成器难以在长期内保持场景的一致性。为了解决这些限制，我们引入了基于3D表面元素（surfels）几何索引过去视图的视图记忆机制（VMem）。通过这种方式，当生成新视图时，可以高效地检索最相关的过去视图。我们的方法仅关注这些相关视图，在保持计算成本较低的同时，能够产生连贯性较强的想象环境探索结果。 我们在具有挑战性的长期场景合成基准测试中评估了这种方法，并展示了在维持场景一致性和相机控制方面优于现有方法的性能。
URL
https://arxiv.org/abs/2506.18903
PDF
-
TAMMs: Temporal-Aware Multimodal Model for Satellite Image Change Understanding and Forecasting
2025-06-23 17:26:16Zhongbin Guo, Yuhao Wang, Ping Jian, Xinyue Chen, Wei Peng, Ertai EAbstract
Satellite image time-series analysis demands fine-grained spatial-temporal reasoning, which remains a challenge for existing multimodal large language models (MLLMs). In this work, we study the capabilities of MLLMs on a novel task that jointly targets temporal change understanding and future scene generation, aiming to assess their potential for modeling complex multimodal dynamics over time. We propose TAMMs, a Temporal-Aware Multimodal Model for satellite image change understanding and forecasting, which enhances frozen MLLMs with lightweight temporal modules for structured sequence encoding and contextual prompting. To guide future image generation, TAMMs introduces a Semantic-Fused Control Injection (SFCI) mechanism that adaptively combines high-level semantic reasoning and structural priors within an enhanced ControlNet. This dual-path conditioning enables temporally consistent and semantically grounded image synthesis. Experiments demonstrate that TAMMs outperforms strong MLLM baselines in both temporal change understanding and future image forecasting tasks, highlighting how carefully designed temporal reasoning and semantic fusion can unlock the full potential of MLLMs for spatio-temporal understanding.
Abstract (translated)
卫星图像时间序列分析需要精细的时空推理，这对现有的多模态大型语言模型（MLLM）来说仍然是一个挑战。在本项工作中，我们研究了这些模型在一个新颖的任务上的能力，该任务旨在同时理解时间变化并生成未来场景，以评估它们建模复杂多模态动态变化随时间演变的能力。 为此，我们提出了TAMMs，即时空感知的多模态模型（Temporal-Aware Multimodal Model），用于卫星图像变化理解和预测。通过为冻结的MLLM添加轻量级的时间模块，该模型能够进行结构化序列编码和上下文提示增强。为了指导未来图像生成，TAMMs引入了语义融合控制注入机制（Semantic-Fused Control Injection, SFCI）。这种机制在改进后的ControlNet中适应性地结合高级别的语义推理和结构性先验知识。双路径条件处理使得图像合成既具有时间一致性又基于语义。 实验结果表明，TAMMs在时间和变化理解以及未来图像预测任务上都超过了强大的MLLM基准模型，这突显了精心设计的时间推理和语义融合如何解锁MLLM在时空理解方面的全部潜力。
URL
https://arxiv.org/abs/2506.18862
PDF
-
Long-term Traffic Simulation with Interleaved Autoregressive Motion and Scenario Generation
2025-06-20 17:59:21Xiuyu Yang, Shuhan Tan, Philipp Kr\"ahenb\"uhlAbstract
An ideal traffic simulator replicates the realistic long-term point-to-point trip that a self-driving system experiences during deployment. Prior models and benchmarks focus on closed-loop motion simulation for initial agents in a scene. This is problematic for long-term simulation. Agents enter and exit the scene as the ego vehicle enters new regions. We propose InfGen, a unified next-token prediction model that performs interleaved closed-loop motion simulation and scene generation. InfGen automatically switches between closed-loop motion simulation and scene generation mode. It enables stable long-term rollout simulation. InfGen performs at the state-of-the-art in short-term (9s) traffic simulation, and significantly outperforms all other methods in long-term (30s) simulation. The code and model of InfGen will be released at this https URL
Abstract (translated)
一个理想的交通模拟器能够复制自主驾驶系统在部署期间经历的真实长期点到点的旅程。之前的模型和基准测试主要集中在场景中初始代理的闭环运动仿真上，这不利于长期仿真的实施。随着主车辆进入新的区域，代理会不断进出场景，这就造成了问题。 我们提出了一种名为InfGen的统一模型，它可以执行交错进行的闭环运动模拟与场景生成任务。InfGen能够自动在闭环运动模拟和场景生成模式之间切换，从而实现稳定的长时间仿真流程。该模型在短期（9秒）交通模拟方面达到了当前最佳水平，并且在长期（30秒）模拟中显著超越了所有其他方法。 InfGen的代码和模型将在以下链接发布：[请参见原文链接以获取正确的URL]
URL
https://arxiv.org/abs/2506.17213
PDF
-
DreamCube: 3D Panorama Generation via Multi-plane Synchronization
2025-06-20 17:55:06Yukun Huang, Yanning Zhou, Jianan Wang, Kaiyi Huang, Xihui LiuAbstract
3D panorama synthesis is a promising yet challenging task that demands high-quality and diverse visual appearance and geometry of the generated omnidirectional content. Existing methods leverage rich image priors from pre-trained 2D foundation models to circumvent the scarcity of 3D panoramic data, but the incompatibility between 3D panoramas and 2D single views limits their effectiveness. In this work, we demonstrate that by applying multi-plane synchronization to the operators from 2D foundation models, their capabilities can be seamlessly extended to the omnidirectional domain. Based on this design, we further introduce DreamCube, a multi-plane RGB-D diffusion model for 3D panorama generation, which maximizes the reuse of 2D foundation model priors to achieve diverse appearances and accurate geometry while maintaining multi-view consistency. Extensive experiments demonstrate the effectiveness of our approach in panoramic image generation, panoramic depth estimation, and 3D scene generation.
Abstract (translated)
360度全景图合成是一项既具前景又充满挑战的任务，它要求生成的全方位内容具有高质量且多样的视觉外观和几何结构。现有方法通过利用预训练2D基础模型中的丰富图像先验来克服3D全景数据稀缺的问题，但3D全景与2D单一视图之间的不兼容性限制了这些方法的有效性。在本工作中，我们展示了通过将2D基础模型的操作符应用于多平面同步技术，可以无缝地扩展它们的能力以适用于全方位领域。基于这一设计，我们进一步提出了DreamCube，这是一种用于生成3D全景图的多平面RGB-D扩散模型，它最大化利用了2D基础模型先验知识来实现多样化的外观和准确的几何结构，并同时保持多视角一致性。广泛的实验展示了我们的方法在全景图像生成、全景深度估计以及三维场景生成方面的有效性。
URL
https://arxiv.org/abs/2506.17206
PDF
-
4Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generation
2025-06-18 23:44:59Chaoyang Wang, Ashkan Mirzaei, Vidit Goel, Willi Menapace, Aliaksandr Siarohin, Avalon Vinella, Michael Vasilkovsky, Ivan Skorokhodov, Vladislav Shakhrai, Sergey Korolev, Sergey Tulyakov, Peter WonkaAbstract
We propose the first framework capable of computing a 4D spatio-temporal grid of video frames and 3D Gaussian particles for each time step using a feed-forward architecture. Our architecture has two main components, a 4D video model and a 4D reconstruction model. In the first part, we analyze current 4D video diffusion architectures that perform spatial and temporal attention either sequentially or in parallel within a two-stream design. We highlight the limitations of existing approaches and introduce a novel fused architecture that performs spatial and temporal attention within a single layer. The key to our method is a sparse attention pattern, where tokens attend to others in the same frame, at the same timestamp, or from the same viewpoint. In the second part, we extend existing 3D reconstruction algorithms by introducing a Gaussian head, a camera token replacement algorithm, and additional dynamic layers and training. Overall, we establish a new state of the art for 4D generation, improving both visual quality and reconstruction capability.
Abstract (translated)
我们提出了一种框架，该框架能够使用前馈架构计算视频帧的四维时空网格和每个时间步的三维高斯粒子。我们的架构主要包含两个部分：一个四维视频模型和一个四维重建模型。 在第一部分中，我们分析了当前的四维视频扩散架构，这些架构在其双流设计中要么顺序地、要么并行地执行空间和时间注意力操作。我们指出现有方法的局限性，并引入了一种新的融合架构，在单层内同时执行空间和时间注意力操作。我们的方法的关键在于一种稀疏注意模式，其中标记物（tokens）可以注意到同一帧内的其他标记物、相同时间戳上的或从相同视角来的标记物。 在第二部分中，我们通过引入高斯头（Gaussian head）、相机标记替换算法以及额外的动态层和训练扩展了现有的三维重建算法。总体而言，我们为四维生成建立了新的业界标准，在视觉质量和重建能力方面都有所提升。
URL
https://arxiv.org/abs/2506.18839
PDF
-
ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxies
2025-06-17 08:50:05Jinyan Yuan, Bangbang Yang, Keke Wang, Panwang Pan, Lin Ma, Xuehai Zhang, Xiao Liu, Zhaopeng Cui, Yuewen MaAbstract
Automatic creation of 3D scenes for immersive VR presence has been a significant research focus for decades. However, existing methods often rely on either high-poly mesh modeling with post-hoc simplification or massive 3D Gaussians, resulting in a complex pipeline or limited visual realism. In this paper, we demonstrate that such exhaustive modeling is unnecessary for achieving compelling immersive experience. We introduce ImmerseGen, a novel agent-guided framework for compact and photorealistic world modeling. ImmerseGen represents scenes as hierarchical compositions of lightweight geometric proxies, i.e., simplified terrain and billboard meshes, and generates photorealistic appearance by synthesizing RGBA textures onto these proxies. Specifically, we propose terrain-conditioned texturing for user-centric base world synthesis, and RGBA asset texturing for midground and foreground this http URL reformulation offers several advantages: (i) it simplifies modeling by enabling agents to guide generative models in producing coherent textures that integrate seamlessly with the scene; (ii) it bypasses complex geometry creation and decimation by directly synthesizing photorealistic textures on proxies, preserving visual quality without degradation; (iii) it enables compact representations suitable for real-time rendering on mobile VR headsets. To automate scene creation from text prompts, we introduce VLM-based modeling agents enhanced with semantic grid-based analysis for improved spatial reasoning and accurate asset placement. ImmerseGen further enriches scenes with dynamic effects and ambient audio to support multisensory immersion. Experiments on scene generation and live VR showcases demonstrate that ImmerseGen achieves superior photorealism, spatial coherence and rendering efficiency compared to prior methods. Project webpage: this https URL.
Abstract (translated)
几十年来，自动创建用于沉浸式VR体验的3D场景一直是重要的研究焦点。然而，现有的方法通常依赖于高多边形网格建模后再进行简化处理或使用大量的三维高斯模型，这导致了复杂的流程或是有限的真实感视觉效果。在本文中，我们展示了为了实现令人信服的沉浸式体验，并不需要这种详尽的建模工作。我们引入了ImmerseGen，这是一个新的代理引导框架，用于紧凑且逼真的世界建模。 ImmerseGen将场景表示为轻量级几何代理（即简化的地形和海报网格）的层次组合，并通过合成RGBA纹理在这些代理上生成逼真的外观效果。具体而言，我们提出了基于地形条件的纹理处理方法来合成用户为中心的基本世界的组成元素，以及用于中景和前景元素的RGBA资产纹理化。这种重新构想提供了几个优势：(i) 它简化了建模过程，通过让代理指导生成模型生产与场景无缝集成的连贯纹理；(ii) 无需复杂的几何创建和减面处理，直接在代理上合成逼真的纹理可以保持视觉质量而不退化；(iii) 支持适合移动VR头显实时渲染的紧凑表示形式。 为了从文本提示中自动创建场景，我们引入了增强有语义网格分析功能的VLM建模代理，以改进空间推理和准确的资产放置。ImmerseGen进一步通过动态效果和环境音频来丰富场景，支持多感官沉浸体验。实验结果表明，在场景生成和现场VR演示中，与先前方法相比，ImmerseGen在逼真度、空间一致性以及渲染效率方面都表现出色。 项目网页：[请参阅原文提供的链接]
URL
https://arxiv.org/abs/2506.14315
PDF
-
X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability
2025-06-16 14:43:18Yu Yang, Alan Liang, Jianbiao Mei, Yukai Ma, Yong Liu, Gim Hee LeeAbstract
Diffusion models are advancing autonomous driving by enabling realistic data synthesis, predictive end-to-end planning, and closed-loop simulation, with a primary focus on temporally consistent generation. However, the generation of large-scale 3D scenes that require spatial coherence remains underexplored. In this paper, we propose X-Scene, a novel framework for large-scale driving scene generation that achieves both geometric intricacy and appearance fidelity, while offering flexible controllability. Specifically, X-Scene supports multi-granular control, including low-level conditions such as user-provided or text-driven layout for detailed scene composition and high-level semantic guidance such as user-intent and LLM-enriched text prompts for efficient customization. To enhance geometrical and visual fidelity, we introduce a unified pipeline that sequentially generates 3D semantic occupancy and the corresponding multiview images, while ensuring alignment between modalities. Additionally, we extend the generated local region into a large-scale scene through consistency-aware scene outpainting, which extrapolates new occupancy and images conditioned on the previously generated area, enhancing spatial continuity and preserving visual coherence. The resulting scenes are lifted into high-quality 3DGS representations, supporting diverse applications such as scene exploration. Comprehensive experiments demonstrate that X-Scene significantly advances controllability and fidelity for large-scale driving scene generation, empowering data generation and simulation for autonomous driving.
Abstract (translated)
扩散模型通过实现现实数据合成、预测端到端规划和闭环模拟，在推动自动驾驶技术进步方面发挥了重要作用，尤其是在时间上的一致性生成方面。然而，对于需要空间连贯性的大规模3D场景的生成研究仍较为有限。为此，本文提出了一种名为X-Scene的新框架，用于进行大规模驾驶场景生成，该框架能够同时实现几何复杂性和外观真实性，并提供灵活的可控性。 具体而言，X-Scene支持多粒度控制，包括低级条件（如用户提供的或文本驱动的布局），以构成详细的场景组合；以及高级语义指导（如用户的意图和增强型LLM文本提示）用于高效定制。为了提升几何学和视觉真实感，我们引入了一个统一的管道，在该管道中顺序生成3D语义占用图和对应的多视角图像，并确保不同模式之间的一致性。此外，通过一致性感知场景扩增技术，我们将生成的局部区域扩展为大规模场景，以增强空间连续性和保持视觉连贯性。 最终生成的场景被提升至高质量的3DGS表示形式，支持如场景探索等多样化的应用需求。全面的实验表明，X-Scene显著提升了大规模驾驶场景生成中的可控性和真实感，从而增强了自动驾驶数据生成和模拟的能力。
URL
https://arxiv.org/abs/2506.13558
PDF
-
EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligence
2025-06-12 11:43:50Wang Xinjie, Liu Liu, Cao Yu, Wu Ruiqi, Qin Wenkang, Wang Dehui, Sui Wei, Su ZhizhongAbstract
Constructing a physically realistic and accurately scaled simulated 3D world is crucial for the training and evaluation of embodied intelligence tasks. The diversity, realism, low cost accessibility and affordability of 3D data assets are critical for achieving generalization and scalability in embodied AI. However, most current embodied intelligence tasks still rely heavily on traditional 3D computer graphics assets manually created and annotated, which suffer from high production costs and limited realism. These limitations significantly hinder the scalability of data driven approaches. We present EmbodiedGen, a foundational platform for interactive 3D world generation. It enables the scalable generation of high-quality, controllable and photorealistic 3D assets with accurate physical properties and real-world scale in the Unified Robotics Description Format (URDF) at low cost. These assets can be directly imported into various physics simulation engines for fine-grained physical control, supporting downstream tasks in training and evaluation. EmbodiedGen is an easy-to-use, full-featured toolkit composed of six key modules: Image-to-3D, Text-to-3D, Texture Generation, Articulated Object Generation, Scene Generation and Layout Generation. EmbodiedGen generates diverse and interactive 3D worlds composed of generative 3D assets, leveraging generative AI to address the challenges of generalization and evaluation to the needs of embodied intelligence related research. Code is available at this https URL.
Abstract (translated)
构建一个物理上真实且比例准确的三维模拟世界对于具身智能任务的训练和评估至关重要。三维数据资产的多样性、现实性、低成本获取性和可负担性是实现具身人工智能中的泛化和规模扩展的关键因素。然而，目前大多数具身智能任务仍然严重依赖于传统的由人工创建和标注的3D计算机图形资源，这些资源面临着高昂的生产成本和有限的真实感问题。这些问题显著限制了数据驱动方法的可扩展性。 我们提出了一种名为EmbodiedGen的基础平台，该平台用于生成交互式三维世界。它能够在低成本的情况下，大规模地生成高质量、可控且高度逼真的3D资产，并具备准确的物理特性和实际世界的规模（在统一机器人描述格式URDF中）。这些资源可以直接导入到各种物理模拟引擎中以实现细微程度的物理控制，支持下游任务中的训练和评估工作。EmbodiedGen是一个易于使用的全功能工具包，由六个关键模块组成：Image-to-3D、Text-to-3D、Texture Generation（纹理生成）、Articulated Object Generation（连杆对象生成）、Scene Generation（场景生成）和Layout Generation（布局生成）。通过利用生成式AI技术，EmbodiedGen能够创建包含生成式3D资产的多样化且可交互的三维世界，以解决具身智能相关研究中泛化与评估方面的挑战。该代码可在提供的URL处获取。
URL
https://arxiv.org/abs/2506.10600
PDF
-
PIG: Physically-based Multi-Material Interaction with 3D Gaussians
2025-06-09 11:25:21Zeyu Xiao, Zhenyi Wu, Mingyang Sun, Qipeng Yan, Yufan Guo, Zhuoer Liang, Lihua ZhangAbstract
3D Gaussian Splatting has achieved remarkable success in reconstructing both static and dynamic 3D scenes. However, in a scene represented by 3D Gaussian primitives, interactions between objects suffer from inaccurate 3D segmentation, imprecise deformation among different materials, and severe rendering artifacts. To address these challenges, we introduce PIG: Physically-Based Multi-Material Interaction with 3D Gaussians, a novel approach that combines 3D object segmentation with the simulation of interacting objects in high precision. Firstly, our method facilitates fast and accurate mapping from 2D pixels to 3D Gaussians, enabling precise 3D object-level segmentation. Secondly, we assign unique physical properties to correspondingly segmented objects within the scene for multi-material coupled interactions. Finally, we have successfully embedded constraint scales into deformation gradients, specifically clamping the scaling and rotation properties of the Gaussian primitives to eliminate artifacts and achieve geometric fidelity and visual consistency. Experimental results demonstrate that our method not only outperforms the state-of-the-art (SOTA) in terms of visual quality, but also opens up new directions and pipelines for the field of physically realistic scene generation.
Abstract (translated)
3D高斯点阵（Gaussian Splatting）在重建静态和动态的三维场景方面取得了显著的成功。然而，在由3D高斯原语表示的场景中，物体之间的交互受到不准确的三维分割、不同材料间变形的不精确性以及严重的渲染伪影的影响。为了应对这些挑战，我们引入了PIG：基于物理的多材料相互作用与3D高斯方法（Physically-Based Multi-Material Interaction with 3D Gaussians），这是一种结合了快速且精准地从2D像素映射到3D高斯点阵，并在物体级进行精确三维分割的新方法，同时模拟场景内交互对象之间的复杂物理现象。具体而言： 首先，我们的方法支持将二维像素快速、准确地映射为3D高斯，从而实现物体级别的精细三维分割。 其次，在场景中我们为不同材料的对应分割对象分配特定的物理属性，以实现多材料耦合互动。 最后，我们将约束尺度成功嵌入到变形梯度中，特别针对高斯原语的缩放和旋转特性进行了限制，这有助于消除伪影，并达到几何精确性和视觉一致性的目标。 实验结果表明，我们的方法不仅在视觉质量上超过了最先进的技术（SOTA），而且还为物理现实场景生成领域开辟了新的方向和技术途径。
URL
https://arxiv.org/abs/2506.07657
PDF
-
Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency
2025-06-09 07:20:49Xiangyu Guo, Zhanqian Wu, Kaixin Xiong, Ziyang Xu, Lijun Zhou, Gangwei Xu, Shaoqing Xu, Haiyang Sun, Bing Wang, Guang Chen, Hangjun Ye, Wenyu Liu, Xinggang WangAbstract
We present Genesis, a unified framework for joint generation of multi-view driving videos and LiDAR sequences with spatio-temporal and cross-modal consistency. Genesis employs a two-stage architecture that integrates a DiT-based video diffusion model with 3D-VAE encoding, and a BEV-aware LiDAR generator with NeRF-based rendering and adaptive sampling. Both modalities are directly coupled through a shared latent space, enabling coherent evolution across visual and geometric domains. To guide the generation with structured semantics, we introduce DataCrafter, a captioning module built on vision-language models that provides scene-level and instance-level supervision. Extensive experiments on the nuScenes benchmark demonstrate that Genesis achieves state-of-the-art performance across video and LiDAR metrics (FVD 16.95, FID 4.24, Chamfer 0.611), and benefits downstream tasks including segmentation and 3D detection, validating the semantic fidelity and practical utility of the generated data.
Abstract (translated)
我们介绍了Genesis，这是一个统一的框架，用于生成多视角驾驶视频和LiDAR序列，并确保它们在时空上和跨模态上的连贯性。Genesis采用两阶段架构，结合了基于DiT（Diffusion Transformer）的视频扩散模型与3D-VAE编码器，以及使用NeRF渲染和自适应采样的BEV感知LiDAR生成器。两种模式通过共享潜在空间直接耦合在一起，从而实现在视觉和几何领域的一致演变。为了用结构化的语义引导生成过程，我们引入了DataCrafter，这是一个基于视觉-语言模型的图像描述模块，能够提供场景级和实例级监督。在nuScenes基准测试上的大量实验表明，Genesis在视频和LiDAR指标（FVD 16.95、FID 4.24、Chamfer 0.611）上均达到了最先进的性能，并且增强了下游任务如分割和3D检测的实用性，验证了生成数据在语义保真度和实际应用中的价值。
URL
https://arxiv.org/abs/2506.07497
PDF
-
SceneLCM: End-to-End Layout-Guided Interactive Indoor Scene Generation with Latent Consistency Model
2025-06-08 11:30:31Yangkai Lin, Jiabao Lei, Kui JiaAbstract
Our project page: this https URL. Automated generation of complex, interactive indoor scenes tailored to user prompt remains a formidable challenge. While existing methods achieve indoor scene synthesis, they struggle with rigid editing constraints, physical incoherence, excessive human effort, single-room limitations, and suboptimal material quality. To address these limitations, we propose SceneLCM, an end-to-end framework that synergizes Large Language Model (LLM) for layout design with Latent Consistency Model(LCM) for scene optimization. Our approach decomposes scene generation into four modular pipelines: (1) Layout Generation. We employ LLM-guided 3D spatial reasoning to convert textual descriptions into parametric blueprints(3D layout). And an iterative programmatic validation mechanism iteratively refines layout parameters through LLM-mediated dialogue loops; (2) Furniture Generation. SceneLCM employs Consistency Trajectory Sampling(CTS), a consistency distillation sampling loss guided by LCM, to form fast, semantically rich, and high-quality representations. We also offer two theoretical justification to demonstrate that our CTS loss is equivalent to consistency loss and its distillation error is bounded by the truncation error of the Euler solver; (3) Environment Optimization. We use a multiresolution texture field to encode the appearance of the scene, and optimize via CTS loss. To maintain cross-geometric texture coherence, we introduce a normal-aware cross-attention decoder to predict RGB by cross-attending to the anchors locations in geometrically heterogeneous instance. (4)Physically Editing. SceneLCM supports physically editing by integrating physical simulation, achieved persistent physical realism. Extensive experiments validate SceneLCM's superiority over state-of-the-art techniques, showing its wide-ranging potential for diverse applications.
Abstract (translated)
我们的项目页面：此 HTTPS URL。自动生成复杂且互动性强的室内场景以响应用户提示仍然是一个重大挑战。虽然现有方法能够合成室内场景，但它们在编辑约束、物理不一致性、人力投入过大、仅限单个房间以及材料质量不佳等方面存在局限性。为了解决这些问题，我们提出了SceneLCM框架，该框架整合了大型语言模型（LLM）用于布局设计和潜在一致模型（LCM）用于场景优化的端到端方法。 我们的方法将场景生成分解成四个模块化管道： 1. **布局生成**：我们使用由LLM引导的3D空间推理技术，将文本描述转换为参数化的蓝图（即3D布局）。同时，通过LLM中介对话循环进行迭代验证机制，逐步细化布局参数。 2. **家具生成**：SceneLCM采用了一致性轨迹采样（CTS）方法，该方法由LCM指导的一致性蒸馏采样损失驱动，能够快速形成语义丰富且高质量的表示。此外，我们提供了两个理论依据来证明我们的CTS损失等同于一致性损失，并且其蒸馏误差被欧拉解算器截断误差所限定。 3. **环境优化**：我们使用多分辨率纹理字段编码场景外观，并通过CTS损失进行优化。为了保持跨几何结构的纹理连贯性，我们引入了一种法线感知交叉注意解码器，通过交叉关注不同实例中的锚定位置来预测RGB值。 4. **物理编辑支持**：SceneLCM通过整合物理模拟实现了持久的物理真实性，从而支持场景的物理编辑。广泛实验验证了SceneLCM在当前技术前沿上的优越性，并展示了其在各种应用领域的广泛应用潜力。
URL
https://arxiv.org/abs/2506.07091
PDF
-
AI-powered Contextual 3D Environment Generation: A Systematic Review
2025-06-05 15:56:28Miguel Silva, Alexandre Valle de CarvalhoAbstract
The generation of high-quality 3D environments is crucial for industries such as gaming, virtual reality, and cinema, yet remains resource-intensive due to the reliance on manual processes. This study performs a systematic review of existing generative AI techniques for 3D scene generation, analyzing their characteristics, strengths, limitations, and potential for improvement. By examining state-of-the-art approaches, it presents key challenges such as scene authenticity and the influence of textual inputs. Special attention is given to how AI can blend different stylistic domains while maintaining coherence, the impact of training data on output quality, and the limitations of current models. In addition, this review surveys existing evaluation metrics for assessing realism and explores how industry professionals incorporate AI into their workflows. The findings of this study aim to provide a comprehensive understanding of the current landscape and serve as a foundation for future research on AI-driven 3D content generation. Key findings include that advanced generative architectures enable high-quality 3D content creation at a high computational cost, effective multi-modal integration techniques like cross-attention and latent space alignment facilitate text-to-3D tasks, and the quality and diversity of training data combined with comprehensive evaluation metrics are critical to achieving scalable, robust 3D scene generation.
Abstract (translated)
高质量的三维环境生成对于游戏、虚拟现实和电影等行业至关重要，但因其依赖于手动过程而仍然非常耗费资源。本研究对现有的用于生成3D场景的生成式AI技术进行了系统性回顾，分析了这些技术的特点、优势、局限性和改进潜力。通过考察最前沿的方法，它揭示了一些关键挑战，如场景的真实性以及文本输入的影响。特别关注的是，如何利用AI在保持一致性的同时融合不同的风格领域，训练数据对输出质量的影响以及当前模型的限制。此外，本综述还调研了现有的评估真实性的指标，并探讨了行业专业人士如何将AI整合到他们的工作流程中。这项研究的发现旨在为当前的研究提供一个全面的理解基础，并作为未来关于基于人工智能的3D内容生成研究的基础。 关键发现包括：先进的生成架构能够在高计算成本下创建高质量的3D内容；有效的多模态集成技术，如跨注意力和潜在空间对齐，有助于文本到3D的任务；以及高质量且多样化的训练数据结合全面的评估指标对于实现可扩展、稳健的3D场景生成至关重要。
URL
https://arxiv.org/abs/2506.05449
PDF
-
Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation
2025-06-04 17:59:04Tianyu Huang, Wangguandong Zheng, Tengfei Wang, Yuhao Liu, Zhenwei Wang, Junta Wu, Jie Jiang, Hui Li, Rynson W. H. Lau, Wangmeng Zuo, Chunchao GuoAbstract
Real-world applications like video gaming and virtual reality often demand the ability to model 3D scenes that users can explore along custom camera trajectories. While significant progress has been made in generating 3D objects from text or images, creating long-range, 3D-consistent, explorable 3D scenes remains a complex and challenging problem. In this work, we present Voyager, a novel video diffusion framework that generates world-consistent 3D point-cloud sequences from a single image with user-defined camera path. Unlike existing approaches, Voyager achieves end-to-end scene generation and reconstruction with inherent consistency across frames, eliminating the need for 3D reconstruction pipelines (e.g., structure-from-motion or multi-view stereo). Our method integrates three key components: 1) World-Consistent Video Diffusion: A unified architecture that jointly generates aligned RGB and depth video sequences, conditioned on existing world observation to ensure global coherence 2) Long-Range World Exploration: An efficient world cache with point culling and an auto-regressive inference with smooth video sampling for iterative scene extension with context-aware consistency, and 3) Scalable Data Engine: A video reconstruction pipeline that automates camera pose estimation and metric depth prediction for arbitrary videos, enabling large-scale, diverse training data curation without manual 3D annotations. Collectively, these designs result in a clear improvement over existing methods in visual quality and geometric accuracy, with versatile applications.
Abstract (translated)
现实世界的应用，如视频游戏和虚拟现实，经常需要能够根据用户的自定义相机轨迹来模拟可以探索的三维场景。虽然从文本或图像生成3D物体已取得显著进展，但创建长距离、一致且可探索的3D场景仍然是一个复杂且具有挑战性的问题。在这项工作中，我们介绍了Voyager，这是一种新颖的视频扩散框架，可以从单张图片和用户定义的相机路径中生成世界一致性3D点云序列。与现有方法不同，Voyager实现了端到端的场景生成与重建，并在整个帧之间保持固有的连贯性，从而消除了对三维重建管线（例如运动结构或多视图立体匹配）的需求。 我们的方法集成了三个关键组件： 1. **世界一致性视频扩散**：一种统一架构，在给定现有世界观察的基础上联合生成一致的RGB和深度视频序列，以确保全局连贯性。 2. **长距离世界探索**：一个高效的世界缓存系统与点剔除结合，并通过平滑视频采样实现了自回归推理，能够进行上下文感知的一致迭代场景扩展。 3. **可扩展数据引擎**：一种视频重建管线，自动估计相机姿态并预测任意视频的度量深度，使大规模、多样化的训练数据整理成为可能，而无需手动标注三维信息。 通过这些设计，Voyager在视觉质量和几何精度上相对于现有方法有了明显的改进，并具有广泛的应用潜力。
URL
https://arxiv.org/abs/2506.04225
PDF
-
Generalized Diffusion Detector: Mining Robust Features from Diffusion Models for Domain-Generalized Detection
2025-06-04 16:55:35Boyong He, Yuxiang Ji, Qianwen Ye, Zhuoyue Tan, Liaoni WuAbstract
Domain generalization (DG) for object detection aims to enhance detectors' performance in unseen scenarios. This task remains challenging due to complex variations in real-world applications. Recently, diffusion models have demonstrated remarkable capabilities in diverse scene generation, which inspires us to explore their potential for improving DG tasks. Instead of generating images, our method extracts multi-step intermediate features during the diffusion process to obtain domain-invariant features for generalized detection. Furthermore, we propose an efficient knowledge transfer framework that enables detectors to inherit the generalization capabilities of diffusion models through feature and object-level alignment, without increasing inference time. We conduct extensive experiments on six challenging DG benchmarks. The results demonstrate that our method achieves substantial improvements of 14.0% mAP over existing DG approaches across different domains and corruption types. Notably, our method even outperforms most domain adaptation methods without accessing any target domain data. Moreover, the diffusion-guided detectors show consistent improvements of 15.9% mAP on average compared to the baseline. Our work aims to present an effective approach for domain-generalized detection and provide potential insights for robust visual recognition in real-world scenarios. The code is available at this https URL.
Abstract (translated)
领域泛化（Domain Generalization，DG）在目标检测中的应用旨在提高检测器在未见过场景下的表现。由于实际应用场景中存在复杂多变的情况，这一任务依然充满挑战。最近，扩散模型展示了其在生成多样化场景方面的卓越能力，这激发了我们将它们用于改善DG任务的潜力的研究兴趣。与直接生成图像不同，我们的方法通过在扩散过程中提取多步中间特征来获取跨域不变性特征，从而实现泛化检测。 此外，我们提出了一种高效的知识转移框架，使检测器能够继承扩散模型的泛化能力，而无需增加推理时间，并且该框架是通过特征级和对象级对齐完成的。我们在六个具有挑战性的DG基准数据集上进行了广泛的实验，结果显示我们的方法在不同域和损坏类型上比现有DG方法提高了14.0% mAP（平均精度）。值得注意的是，即使不访问目标域数据，我们提出的方法也超越了大多数领域适应方法的表现。 此外，由扩散模型指导的检测器相比基准线，在所有测试场景中平均mAP值提升达到了15.9%。我们的工作旨在提供一种有效的跨领域泛化检测解决方案，并为在实际场景中的稳健视觉识别提供潜在见解。代码可以在以下链接获取：[此链接](https://this https URL)。 请注意，最后一个URL应替换为您实际提供的代码仓库地址或其他相关资源的正确链接。
URL
https://arxiv.org/abs/2503.02101
PDF
-
FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts
2025-06-03 12:01:41Tongyuan Bai, Wangyuanfan Bai, Dong Chen, Tieru Wu, Manyi Li, Rui MaAbstract
Controllability plays a crucial role in the practical applications of 3D indoor scene synthesis. Existing works either allow rough language-based control, that is convenient but lacks fine-grained scene customization, or employ graph based control, which offers better controllability but demands considerable knowledge for the cumbersome graph design process. To address these challenges, we present FreeScene, a user-friendly framework that enables both convenient and effective control for indoor scene this http URL, FreeScene supports free-form user inputs including text description and/or reference images, allowing users to express versatile design intentions. The user inputs are adequately analyzed and integrated into a graph representation by a VLM-based Graph Designer. We then propose MG-DiT, a Mixed Graph Diffusion Transformer, which performs graph-aware denoising to enhance scene generation. Our MG-DiT not only excels at preserving graph structure but also offers broad applicability to various tasks, including, but not limited to, text-to-scene, graph-to-scene, and rearrangement, all within a single model. Extensive experiments demonstrate that FreeScene provides an efficient and user-friendly solution that unifies text-based and graph based scene synthesis, outperforming state-of-the-art methods in terms of both generation quality and controllability in a range of applications.
Abstract (translated)
可控性在3D室内场景合成的实际应用中起着关键作用。现有研究要么提供粗糙的语言控制，虽然使用方便但缺乏精细的场景定制；要么采用基于图的控制方式，这种控制方式提供了更好的可控性，但需要用户具备复杂且繁琐的图形设计知识。为了解决这些问题，我们提出了一种名为FreeScene的框架，它既便捷又有效，能够使用户对室内场景进行细致的控制。 在FreeScene中，支持自由形式的用户输入，包括文本描述和/或参考图像，允许用户表达各种设计意图。通过基于视觉语言模型（VLM）的图设计器，用户的这些输入会被充分分析并整合到一个图形表示之中。然后我们提出了一种混合图扩散变压器(MG-DiT)，该变压器能够执行图感知去噪以增强场景生成过程。我们的MG-DiT不仅擅长保持图形结构，还具有广泛的应用性，包括但不限于文本到场景、图到场景以及重排任务，并且所有这些都在单一模型中实现。 通过广泛的实验验证，FreeScene提供了一种高效且用户友好的解决方案，能够统一基于文本和基于图的场景合成方法。在多种应用场合下，该方案不仅在生成质量上超越了当前最先进的方法，在可控性方面也表现出色。
URL
https://arxiv.org/abs/2506.02781
PDF
-
WorldExplorer: Towards Generating Fully Navigable 3D Scenes
2025-06-02 15:41:31Manuel-Andreas Schneider, Lukas H\"ollein, Matthias Nie{\ss}nerAbstract
Generating 3D worlds from text is a highly anticipated goal in computer vision. Existing works are limited by the degree of exploration they allow inside of a scene, i.e., produce streched-out and noisy artifacts when moving beyond central or panoramic perspectives. To this end, we propose WorldExplorer, a novel method based on autoregressive video trajectory generation, which builds fully navigable 3D scenes with consistent visual quality across a wide range of viewpoints. We initialize our scenes by creating multi-view consistent images corresponding to a 360 degree panorama. Then, we expand it by leveraging video diffusion models in an iterative scene generation pipeline. Concretely, we generate multiple videos along short, pre-defined trajectories, that explore the scene in depth, including motion around objects. Our novel scene memory conditions each video on the most relevant prior views, while a collision-detection mechanism prevents degenerate results, like moving into objects. Finally, we fuse all generated views into a unified 3D representation via 3D Gaussian Splatting optimization. Compared to prior approaches, WorldExplorer produces high-quality scenes that remain stable under large camera motion, enabling for the first time realistic and unrestricted exploration. We believe this marks a significant step toward generating immersive and truly explorable virtual 3D environments.
Abstract (translated)
从文本生成三维世界是计算机视觉领域一个备受期待的目标。现有的方法在场景内部探索方面受到限制，即超出中心或全景视角时会产生拉伸和噪点等不理想的效果。为此，我们提出了WorldExplorer这一新方法，它基于自回归视频轨迹生成技术，在广泛的视角范围内构建出完全可导航且具有连贯视觉质量的三维场景。我们的方法首先通过创建一系列多视图一致性图像（对应360度全景）来初始化场景。然后利用视频扩散模型在迭代场景生成流水线中扩展这些初始设置，具体来说，我们在预定义的短轨迹上生成多个视频，深入探索场景，包括围绕物体运动的情况。我们提出的新颖场景记忆机制让每个视频都基于最相关的先前视图条件化处理，同时一个碰撞检测机制防止了诸如移动进物体等退化的结果产生。最后，通过3D高斯点绘优化将所有生成的视角融合为统一的三维表示。 相比于之前的方法，WorldExplorer能够生成在大范围相机运动下依然保持稳定的高质量场景，首次实现了真实且不受限的探索。我们认为这标志着向生成沉浸式和真正可探索的虚拟三维环境迈出的重要一步。
URL
https://arxiv.org/abs/2506.01799
PDF
