Scene_Generation
-
Scene Splatter: Momentum 3D Scene Generation from Single Image with Video Diffusion Model
2025-04-03 17:00:44Shengjun Zhang, Jinzhao Li, Xin Fei, Hao Liu, Yueqi DuanAbstract
In this paper, we propose Scene Splatter, a momentum-based paradigm for video diffusion to generate generic scenes from single image. Existing methods, which employ video generation models to synthesize novel views, suffer from limited video length and scene inconsistency, leading to artifacts and distortions during further reconstruction. To address this issue, we construct noisy samples from original features as momentum to enhance video details and maintain scene consistency. However, for latent features with the perception field that spans both known and unknown regions, such latent-level momentum restricts the generative ability of video diffusion in unknown regions. Therefore, we further introduce the aforementioned consistent video as a pixel-level momentum to a directly generated video without momentum for better recovery of unseen regions. Our cascaded momentum enables video diffusion models to generate both high-fidelity and consistent novel views. We further finetune the global Gaussian representations with enhanced frames and render new frames for momentum update in the next step. In this manner, we can iteratively recover a 3D scene, avoiding the limitation of video length. Extensive experiments demonstrate the generalization capability and superior performance of our method in high-fidelity and consistent scene generation.
Abstract (translated)
在这篇论文中，我们提出了一种名为Scene Splatter的基于动量的范式，用于从单张图像生成通用场景。现有方法使用视频生成模型来合成新颖视图时，会受到视频长度有限和场景不一致的影响，从而导致在进一步重建过程中出现伪影和失真。为了解决这个问题，我们构建了以原始特征为基础的噪声样本作为动量，以此增强视频细节并保持场景一致性。然而，在具有感知范围涵盖已知区域与未知区域的潜在特征的情况下，这种潜在层次上的动量限制了视频扩散模型在未知区域内生成的能力。因此，我们进一步将前述一致性的视频引入为像素级别的动量，以改善没有使用动量直接生成的视频中未见区域的恢复效果。我们的级联动量使视频扩散模型能够生成既高保真又具有一致性的新颖视图。此外，我们将全局高斯表示进行精细调整，并利用增强后的帧渲染新帧来更新下一步中的动量。通过这种方式，我们可以迭代地恢复一个3D场景，避免了视频长度的限制。广泛的实验展示了我们方法在生成高质量和一致场景方面的泛化能力和卓越性能。
URL
https://arxiv.org/abs/2504.02764
PDF
-
LPA3D: 3D Room-Level Scene Generation from In-the-Wild Images
2025-04-03 07:18:48Ming-Jia Yang, Yu-Xiao Guo, Yang Liu, Bin Zhou, Xin TongAbstract
Generating realistic, room-level indoor scenes with semantically plausible and detailed appearances from in-the-wild images is crucial for various applications in VR, AR, and robotics. The success of NeRF-based generative methods indicates a promising direction to address this challenge. However, unlike their success at the object level, existing scene-level generative methods require additional information, such as multiple views, depth images, or semantic guidance, rather than relying solely on RGB images. This is because NeRF-based methods necessitate prior knowledge of camera poses, which is challenging to approximate for indoor scenes due to the complexity of defining alignment and the difficulty of globally estimating poses from a single image, given the unseen parts behind the camera. To address this challenge, we redefine global poses within the framework of Local-Pose-Alignment (LPA) -- an anchor-based multi-local-coordinate system that uses a selected number of anchors as the roots of these coordinates. Building on this foundation, we introduce LPA-GAN, a novel NeRF-based generative approach that incorporates specific modifications to estimate the priors of camera poses under LPA. It also co-optimizes the pose predictor and scene generation processes. Our ablation study and comparisons with straightforward extensions of NeRF-based object generative methods demonstrate the effectiveness of our approach. Furthermore, visual comparisons with other techniques reveal that our method achieves superior view-to-view consistency and semantic normality.
Abstract (translated)
从野外图像生成具有语义合理且细节丰富的房间级别室内场景对于VR、AR和机器人技术等多种应用至关重要。基于NeRF（神经辐射场）的生成方法的成功表明，这是一个很有前景的方向来解决这一挑战。然而，与它们在物体级别的成功不同，现有的场景级生成方法需要额外的信息，例如多视角图像、深度图或语义指导，而不仅仅是依赖于RGB图像。这是因为基于NeRF的方法需要相机姿态的先验知识，而这对于室内场景来说是具有挑战性的，因为定义对齐的复杂性以及从单张图像全局估计姿态的难度（尤其是考虑到摄像机后方未见的部分）。为了解决这一挑战，我们重新定义了全局姿态，并在局部姿态对齐（Local-Pose-Alignment, LPA）框架内进行操作。这是一个基于锚点的多本地坐标系统，它使用选定数量的锚点作为这些坐标的根。 在此基础上，我们引入了一种新的NeRF生成方法——LPA-GAN，该方法在LPA框架下专门进行了修改以估计相机姿态的先验知识，并且优化了姿态预测和场景生成过程。我们的消融研究以及与基于NeRF的对象生成方法的直接扩展进行比较表明了这种方法的有效性。此外，与其它技术进行的视觉对比显示，我们的方法实现了更好的视图间一致性及语义合理性。 通过这种创新的方法，LPA-GAN能够在仅依赖于RGB图像的情况下有效地生成室内场景，并且在保持高真实性和细节的同时解决了姿态估计的问题，从而为虚拟现实和增强现实中高质量室内外环境建模提供了新的可能。
URL
https://arxiv.org/abs/2504.02337
PDF
-
WonderTurbo: Generating Interactive 3D World in 0.72 Seconds
2025-04-03 04:10:47Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Weijie Wang, Haoyun Li, Guosheng Zhao, Jie Li, Wenkang Qin, Guan Huang, Wenjun MeiAbstract
Interactive 3D generation is gaining momentum and capturing extensive attention for its potential to create immersive virtual experiences. However, a critical challenge in current 3D generation technologies lies in achieving real-time interactivity. To address this issue, we introduce WonderTurbo, the first real-time interactive 3D scene generation framework capable of generating novel perspectives of 3D scenes within 0.72 seconds. Specifically, WonderTurbo accelerates both geometric and appearance modeling in 3D scene generation. In terms of geometry, we propose StepSplat, an innovative method that constructs efficient 3D geometric representations through dynamic updates, each taking only 0.26 seconds. Additionally, we design QuickDepth, a lightweight depth completion module that provides consistent depth input for StepSplat, further enhancing geometric accuracy. For appearance modeling, we develop FastPaint, a 2-steps diffusion model tailored for instant inpainting, which focuses on maintaining spatial appearance consistency. Experimental results demonstrate that WonderTurbo achieves a remarkable 15X speedup compared to baseline methods, while preserving excellent spatial consistency and delivering high-quality output.
Abstract (translated)
交互式3D生成技术正逐渐兴起，并因其创造沉浸式虚拟体验的潜力而备受关注。然而，当前3D生成技术的一个关键挑战在于实现实时互动性。为了解决这一问题，我们推出了WonderTurbo——首个能够实现实时交互式3D场景生成框架，它能够在0.72秒内生成新的3D场景视角。具体而言，WonderTurbo通过加速几何和外观建模来提高3D场景生成的速度。 在几何方面，我们提出了StepSplat方法，这是一种创新的动态更新技术，能够构建高效的3D几何表示，并且每次仅需花费0.26秒。此外，我们设计了QuickDepth轻量级深度完成模块，为StepSplat提供了一致性的深度输入，从而进一步提高几何精度。 在外观建模方面，我们开发了FastPaint——一个专为即时修复绘制而定制的两步扩散模型，专注于保持空间外观的一致性。实验结果表明，与基准方法相比，WonderTurbo实现了15倍的速度提升，并且能够维持出色的空间一致性并输出高质量的结果。
URL
https://arxiv.org/abs/2504.02261
PDF
-
WorldPrompter: Traversable Text-to-Scene Generation
2025-04-02 18:04:32Zhaoyang Zhang, Yannick Hold-Geoffroy, Milo\v{s} Ha\v{s}an, Chen Ziwen, Fujun Luan, Julie Dorsey, Yiwei HuAbstract
Scene-level 3D generation is a challenging research topic, with most existing methods generating only partial scenes and offering limited navigational freedom. We introduce WorldPrompter, a novel generative pipeline for synthesizing traversable 3D scenes from text prompts. We leverage panoramic videos as an intermediate representation to model the 360° details of a scene. WorldPrompter incorporates a conditional 360° panoramic video generator, capable of producing a 128-frame video that simulates a person walking through and capturing a virtual environment. The resulting video is then reconstructed as Gaussian splats by a fast feedforward 3D reconstructor, enabling a true walkable experience within the 3D scene. Experiments demonstrate that our panoramic video generation model achieves convincing view consistency across frames, enabling high-quality panoramic Gaussian splat reconstruction and facilitating traversal over an area of the scene. Qualitative and quantitative results also show it outperforms the state-of-the-art 360° video generators and 3D scene generation models.
Abstract (translated)
场景级别的三维生成是一个具有挑战性的研究课题，大多数现有的方法只能生成部分场景，并且提供有限的导航自由度。我们引入了WorldPrompter，这是一种新颖的生成管道，可以从文本提示中合成可穿越的三维场景。我们利用全景视频作为中间表示来建模场景的360°细节。 WorldPrompter 包含了一个条件性的 360° 全景视频生成器，能够产生一个模拟人在虚拟环境中行走并捕捉环境的128帧视频。然后，通过快速前馈式的三维重建算法将产生的视频重构为高斯点（Gaussian splats），从而在三维场景中实现真正的可行走体验。 实验表明，我们的全景视频生成模型实现了跨帧令人信服的视角一致性，这使得高质量的全景高斯点重建成为可能，并且支持穿越场景中的大片区域。定性和定量的结果也显示它超越了现有的最先进的 360° 视频生成器和三维场景生成模型的表现。 该研究通过结合文本驱动的视频生成与高效的 3D 场景重构技术，为实现高度交互式的虚拟现实体验提供了新的途径。
URL
https://arxiv.org/abs/2504.02045
PDF
-
VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step
2025-04-02 17:59:21Hanyang Wang, Fangfu Liu, Jiawei Chi, Yueqi DuanAbstract
Recovering 3D scenes from sparse views is a challenging task due to its inherent ill-posed problem. Conventional methods have developed specialized solutions (e.g., geometry regularization or feed-forward deterministic model) to mitigate the issue. However, they still suffer from performance degradation by minimal overlap across input views with insufficient visual information. Fortunately, recent video generative models show promise in addressing this challenge as they are capable of generating video clips with plausible 3D structures. Powered by large pretrained video diffusion models, some pioneering research start to explore the potential of video generative prior and create 3D scenes from sparse views. Despite impressive improvements, they are limited by slow inference time and the lack of 3D constraint, leading to inefficiencies and reconstruction artifacts that do not align with real-world geometry structure. In this paper, we propose VideoScene to distill the video diffusion model to generate 3D scenes in one step, aiming to build an efficient and effective tool to bridge the gap from video to 3D. Specifically, we design a 3D-aware leap flow distillation strategy to leap over time-consuming redundant information and train a dynamic denoising policy network to adaptively determine the optimal leap timestep during inference. Extensive experiments demonstrate that our VideoScene achieves faster and superior 3D scene generation results than previous video diffusion models, highlighting its potential as an efficient tool for future video to 3D applications. Project Page: this https URL
Abstract (translated)
从稀疏视角恢复三维场景是一个极具挑战性的任务，因为其本质上是不适定问题。传统的方法开发了专门的解决方案（例如几何正则化或前馈确定性模型）来缓解这些问题。然而，它们仍然在输入视图之间最小重叠且视觉信息不足的情况下遭受性能下降的问题。幸运的是，最近的视频生成模型显示出解决这一挑战的潜力，因为它们能够生成具有合理三维结构的视频片段。借助大规模预训练的视频扩散模型，一些开创性的研究开始探索视频生成先验的潜力，并从稀疏视图中创建三维场景。尽管取得了令人印象深刻的进步，但这些方法受限于推理时间慢以及缺乏三维约束条件，导致效率低下和重建过程中出现与现实世界几何结构不符的艺术瑕疵。 在本文中，我们提出了VideoScene框架，旨在将视频扩散模型精简为一步生成三维场景的方法，目标是建立一个高效且有效的工具，以弥合从视频到三维的差距。具体来说，我们设计了一个3D感知跳跃流精炼策略来跳过耗时冗余信息，并训练动态去噪策略网络在推理过程中自适应地确定最佳跳跃时间步长。 大量的实验表明，我们的VideoScene方法比之前的视频扩散模型更快且更有效地生成三维场景结果，这突显了其作为未来视频到3D应用高效工具的潜力。项目页面：[此URL](https://this-url.com/)（原文中的链接地址请替换为实际链接）。
URL
https://arxiv.org/abs/2504.01956
PDF
-
WorldScore: A Unified Evaluation Benchmark for World Generation
2025-04-01 17:20:23Haoyi Duan, Hong-Xing Yu, Sirui Chen, Li Fei-Fei, Jiajun WuAbstract
We introduce the WorldScore benchmark, the first unified benchmark for world generation. We decompose world generation into a sequence of next-scene generation tasks with explicit camera trajectory-based layout specifications, enabling unified evaluation of diverse approaches from 3D and 4D scene generation to video generation models. The WorldScore benchmark encompasses a curated dataset of 3,000 test examples that span diverse worlds: static and dynamic, indoor and outdoor, photorealistic and stylized. The WorldScore metrics evaluate generated worlds through three key aspects: controllability, quality, and dynamics. Through extensive evaluation of 19 representative models, including both open-source and closed-source ones, we reveal key insights and challenges for each category of models. Our dataset, evaluation code, and leaderboard can be found at this https URL
Abstract (translated)
我们介绍了WorldScore基准测试，这是第一个用于世界生成的统一基准。我们将世界生成分解为一系列基于明确摄像机轨迹布局规范的下一场景生成任务，从而能够对从3D和4D场景生成到视频生成模型等多样方法进行统一评估。WorldScore基准测试包括一个精心策划的数据集，包含3,000个测试示例，涵盖了多样的世界：静态与动态、室内与室外、写实主义和风格化。 WorldScore指标通过三个关键方面来评价生成的世界：可控性、质量和动态特性。通过对19种代表性模型（包括开源和闭源模型）进行广泛的评估，我们揭示了每类模型的关键见解和挑战。我们的数据集、评估代码及排行榜可在此链接获取：[https URL]
URL
https://arxiv.org/abs/2504.00983
PDF
-
Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments
2025-03-28 15:03:41Luke Rowe, Roger Girgis, Anthony Gosselin, Liam Paull, Christopher Pal, Felix HeideAbstract
We introduce Scenario Dreamer, a fully data-driven generative simulator for autonomous vehicle planning that generates both the initial traffic scene - comprising a lane graph and agent bounding boxes - and closed-loop agent behaviours. Existing methods for generating driving simulation environments encode the initial traffic scene as a rasterized image and, as such, require parameter-heavy networks that perform unnecessary computation due to many empty pixels in the rasterized scene. Moreover, we find that existing methods that employ rule-based agent behaviours lack diversity and realism. Scenario Dreamer instead employs a novel vectorized latent diffusion model for initial scene generation that directly operates on the vectorized scene elements and an autoregressive Transformer for data-driven agent behaviour simulation. Scenario Dreamer additionally supports scene extrapolation via diffusion inpainting, enabling the generation of unbounded simulation environments. Extensive experiments show that Scenario Dreamer outperforms existing generative simulators in realism and efficiency: the vectorized scene-generation base model achieves superior generation quality with around 2x fewer parameters, 6x lower generation latency, and 10x fewer GPU training hours compared to the strongest baseline. We confirm its practical utility by showing that reinforcement learning planning agents are more challenged in Scenario Dreamer environments than traditional non-generative simulation environments, especially on long and adversarial driving environments.
Abstract (translated)
我们介绍了Scenario Dreamer，这是一种完全基于数据驱动的自动驾驶车辆规划生成模拟器。它可以生成初始交通场景（包括车道图和代理边框）以及闭环代理行为。现有的用于生成驾驶仿真环境的方法将初始交通场景编码为栅格化图像，并因此需要参数繁重的网络来执行不必要的计算，因为栅格化场景中有许多空白像素。此外，我们发现使用基于规则的代理行为方法缺乏多样性和现实性。 与现有方法不同，Scenario Dreamer采用了一种新颖的向量化潜在扩散模型进行初始场景生成，该模型直接操作向量化的场景元素，并且使用自回归Transformer来进行数据驱动的代理行为模拟。此外，Scenario Dreamer通过扩散填补技术支持场景扩展，从而能够生成不受限的仿真环境。 大量的实验表明，与现有的生成式模拟器相比，Scenario Dreamer在真实性和效率方面表现更佳：向量化场景生成基础模型实现了更高的生成质量，参数数量减少了大约2倍，生成延迟降低了6倍，并且GPU训练时间也减少了10倍。我们通过展示强化学习规划代理在Scenario Dreamer环境中面临更大挑战的事实来确认其实际效用，特别是在长时间和对抗性驾驶环境中的表现优于传统的非生成式仿真环境。
URL
https://arxiv.org/abs/2503.22496
PDF
-
Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistency
2025-03-26 17:59:44Tianqi Liu, Zihao Huang, Zhaoxi Chen, Guangcong Wang, Shoukang Hu, Liao Shen, Huiqiang Sun, Zhiguo Cao, Wei Li, Ziwei LiuAbstract
We present Free4D, a novel tuning-free framework for 4D scene generation from a single image. Existing methods either focus on object-level generation, making scene-level generation infeasible, or rely on large-scale multi-view video datasets for expensive training, with limited generalization ability due to the scarcity of 4D scene data. In contrast, our key insight is to distill pre-trained foundation models for consistent 4D scene representation, which offers promising advantages such as efficiency and generalizability. 1) To achieve this, we first animate the input image using image-to-video diffusion models followed by 4D geometric structure initialization. 2) To turn this coarse structure into spatial-temporal consistent multiview videos, we design an adaptive guidance mechanism with a point-guided denoising strategy for spatial consistency and a novel latent replacement strategy for temporal coherence. 3) To lift these generated observations into consistent 4D representation, we propose a modulation-based refinement to mitigate inconsistencies while fully leveraging the generated information. The resulting 4D representation enables real-time, controllable rendering, marking a significant advancement in single-image-based 4D scene generation.
Abstract (translated)
我们提出了一种名为Free4D的新框架，该框架无需调优即可从单张图像生成4D场景。现有方法要么专注于对象级别的生成，使得场景级别的生成变得不可行；要么依赖于大规模的多视角视频数据集进行昂贵的训练，并且由于缺乏4D场景数据而具有有限的泛化能力。相比之下，我们的关键洞察是通过蒸馏预训练的基础模型来实现一致的4D场景表示，这种方法提供了诸如效率和泛化能力等显著优势。 为了实现这一目标： 1. 我们首先使用图像到视频的扩散模型对输入图像进行动画处理，并随后采用4D几何结构初始化。 2. 为将这种粗略结构转变为时空一致的多视角视频，我们设计了一种自适应引导机制，包括用于空间一致性的一点引导去噪策略和一种新颖的潜在替换策略以实现时间连贯性。 3. 要将这些生成的观察提升到一致的4D表示中，我们提出了一种基于调制的细化方法，通过这种方式可以缓解不一致性，并充分利用生成的信息。 最终得到的4D表示能够实现实时、可控渲染，在单张图像基础上进行4D场景生成方面取得了重大进展。
URL
https://arxiv.org/abs/2503.20785
PDF
-
Global-Local Tree Search for Language Guided 3D Scene Generation
2025-03-24 09:21:13Wei Deng, Mengshi Qi, Huadong MaAbstract
Large Vision-Language Models (VLMs), such as GPT-4, have achieved remarkable success across various fields. However, there are few studies on 3D indoor scene generation with VLMs. This paper considers this task as a planning problem subject to spatial and layout common sense constraints. To solve the problem with a VLM, we propose a new global-local tree search algorithm. Globally, the method places each object sequentially and explores multiple placements during each placement process, where the problem space is represented as a tree. To reduce the depth of the tree, we decompose the scene structure hierarchically, i.e. room level, region level, floor object level, and supported object level. The algorithm independently generates the floor objects in different regions and supported objects placed on different floor objects. Locally, we also decompose the sub-task, the placement of each object, into multiple steps. The algorithm searches the tree of problem space. To leverage the VLM model to produce positions of objects, we discretize the top-down view space as a dense grid and fill each cell with diverse emojis to make to cells distinct. We prompt the VLM with the emoji grid and the VLM produces a reasonable location for the object by describing the position with the name of emojis. The quantitative and qualitative experimental results illustrate our approach generates more plausible 3D scenes than state-of-the-art approaches. Our source code is available at this https URL .
Abstract (translated)
大型视觉-语言模型（VLMs），如GPT-4，在多个领域取得了显著的成功。然而，关于使用VLM生成三维室内场景的研究相对较少。本文将这一任务视为一个受空间和布局常识约束的规划问题，并提出了一种新的全局局部树搜索算法来解决该问题。 在全局层面上，本方法依次放置每个对象并探索每次放置过程中的多种放置方式。为了减少问题空间表示为树结构的深度，我们将场景结构分层分解，即按房间层级、区域层级、楼层物体层级和支持物体层级进行拆解。算法独立生成不同区域中的地面物体和不同地面物体上的支撑物。 在局部层面，我们将每个对象的放置任务进一步细分为多个步骤，并通过探索问题空间的树形图来搜索最优解。为了利用VLM模型产生物体的位置信息，我们把俯视视角的空间离散化为密集网格，并向每个格子填充多样化的表情符号以使其独特可辨认。然后用带有表情符号网格的提示询问VLM，从而得到由表情名称描述的一个合理位置。 通过量化和定性的实验结果表明，我们的方法能够生成比现有最新技术更合理的三维场景。我们的源代码可在提供的链接处获取。
URL
https://arxiv.org/abs/2503.18476
PDF
-
MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse
2025-03-24 09:18:01Zhenyu Pan, Han LiuAbstract
We present MetaSpatial, the first reinforcement learning (RL)-based framework designed to enhance 3D spatial reasoning in vision-language models (VLMs), enabling real-time 3D scene generation without the need for hard-coded optimizations. MetaSpatial addresses two core challenges: (i) the lack of internalized 3D spatial reasoning in VLMs, which limits their ability to generate realistic layouts, and (ii) the inefficiency of traditional supervised fine-tuning (SFT) for layout generation tasks, as perfect ground truth annotations are unavailable. Our key innovation is a multi-turn RL-based optimization mechanism that integrates physics-aware constraints and rendered image evaluations, ensuring generated 3D layouts are coherent, physically plausible, and aesthetically consistent. Methodologically, MetaSpatial introduces an adaptive, iterative reasoning process, where the VLM refines spatial arrangements over multiple turns by analyzing rendered outputs, improving scene coherence progressively. Empirical evaluations demonstrate that MetaSpatial significantly enhances the spatial consistency and formatting stability of various scale models. Post-training, object placements are more realistic, aligned, and functionally coherent, validating the effectiveness of RL for 3D spatial reasoning in metaverse, AR/VR, digital twins, and game development applications. Our code, data, and training pipeline are publicly available at this https URL.
Abstract (translated)
我们介绍了MetaSpatial，这是一个基于强化学习（RL）的框架，旨在增强视觉语言模型（VLMs）中的3D空间推理能力，使其能够在无需硬编码优化的情况下实现实时3D场景生成。MetaSpatial解决了两个核心挑战：(i) VLM中缺乏内在化的3D空间推理能力，这限制了它们生成逼真布局的能力；(ii) 传统监督微调（SFT）方法在布局生成任务中的低效性，因为无法获得完美的地面真实标注。 我们的关键创新是一个基于多轮RL的优化机制，该机制集成了物理感知约束和渲染图像评估，确保生成的3D布局具有连贯性、物理合理性及美学一致性。从方法论的角度来看，MetaSpatial引入了一个自适应迭代推理过程，在这个过程中，VLM通过分析渲染输出来逐步改进空间布置，从而提高场景的一致性。 实证研究表明，MetaSpatial显著增强了各种规模模型的空间一致性和格式稳定性。经过训练后，物体放置变得更加逼真、对齐且功能上连贯，这验证了RL在元宇宙、AR/VR、数字孪生和游戏开发应用中的3D空间推理能力的有效性。我们的代码、数据及培训流程可在以下网址公开获取：[此处插入URL]。
URL
https://arxiv.org/abs/2503.18470
PDF
-
Decorum: A Language-Based Approach For Style-Conditioned Synthesis of Indoor 3D Scenes
2025-03-23 17:48:44Kelly O. Marshall, Omid Poursaeed, Sergiu Oprea, Amit Kumar, Anushrut Jignasu, Chinmay Hegde, Yilei Li, Rakesh RanjanAbstract
3D indoor scene generation is an important problem for the design of digital and real-world environments. To automate this process, a scene generation model should be able to not only generate plausible scene layouts, but also take into consideration visual features and style preferences. Existing methods for this task exhibit very limited control over these attributes, only allowing text inputs in the form of simple object-level descriptions or pairwise spatial relationships. Our proposed method Decorum enables users to control the scene generation process with natural language by adopting language-based representations at each stage. This enables us to harness recent advancements in Large Language Models (LLMs) to model language-to-language mappings. In addition, we show that using a text-based representation allows us to select furniture for our scenes using a novel object retrieval method based on multimodal LLMs. Evaluations on the benchmark 3D-FRONT dataset show that our methods achieve improvements over existing work in text-conditioned scene synthesis and object retrieval.
Abstract (translated)
三维室内场景生成是数字和现实世界环境设计中的一个重要问题。为了实现这一过程的自动化，一个场景生成模型不仅要能够生成合理的场景布局，还应考虑视觉特征和风格偏好。目前的方法在控制这些属性方面表现出非常有限的能力，仅允许使用简单的对象级别描述或空间关系对文本输入进行操作。 我们提出的方法Decorum通过在每个阶段采用基于语言的表示形式，使用户可以用自然语言控制场景生成过程。这使得我们可以利用大型语言模型（LLMs）的最新进展来建模语言到语言的映射。此外，我们还展示了使用文本基表示法可以让我们借助一种基于多模式LLM的新颖对象检索方法为我们的场景选择家具。 在基准数据集3D-FRONT上的评估表明，与现有的以文本条件化场景合成和物体检索的方法相比，我们的方法实现了改进。
URL
https://arxiv.org/abs/2503.18155
PDF
-
SG-Tailor: Inter-Object Commonsense Relationship Reasoning for Scene Graph Manipulation
2025-03-23 09:11:04Haoliang Shang, Hanyu Wu, Guangyao Zhai, Boyang Sun, Fangjinhua Wang, Federico Tombari, Marc PollefeysAbstract
Scene graphs capture complex relationships among objects, serving as strong priors for content generation and manipulation. Yet, reasonably manipulating scene graphs -- whether by adding nodes or modifying edges -- remains a challenging and untouched task. Tasks such as adding a node to the graph or reasoning about a node's relationships with all others are computationally intractable, as even a single edge modification can trigger conflicts due to the intricate interdependencies within the graph. To address these challenges, we introduce SG-Tailor, an autoregressive model that predicts the conflict-free relationship between any two nodes. SG-Tailor not only infers inter-object relationships, including generating commonsense edges for newly added nodes but also resolves conflicts arising from edge modifications to produce coherent, manipulated graphs for downstream tasks. For node addition, the model queries the target node and other nodes from the graph to predict the appropriate relationships. For edge modification, SG-Tailor employs a Cut-And-Stitch strategy to solve the conflicts and globally adjust the graph. Extensive experiments demonstrate that SG-Tailor outperforms competing methods by a large margin and can be seamlessly integrated as a plug-in module for scene generation and robotic manipulation tasks.
Abstract (translated)
场景图捕捉了物体之间的复杂关系，作为内容生成和操作的强大先验知识。然而，合理地操纵场景图（无论是添加节点还是修改边）仍然是一个具有挑战性的未被充分探索的任务。例如，向图中添加一个节点或推理某个节点与其他所有节点的关系在计算上是不可行的，因为即使是单个边的改动也可能由于图内部复杂的相互依赖关系而引发冲突。 为了解决这些挑战，我们提出了SG-Tailor，这是一种自回归模型，它能够预测任意两个节点之间的无冲突关系。SG-Tailor不仅推断了对象间的关系，包括生成新添加节点的常识性边缘连接，而且还解决了由于边修改产生的冲突以产生连贯且可以操作的图供下游任务使用。 对于节点添加，该模型查询目标节点和图中的其他节点来预测适当的关系。 在边修改时，SG-Tailor采用了一种剪切并缝合（Cut-And-Stitch）策略解决冲突，并全局调整整个图形结构。 广泛的实验表明，与竞争方法相比，SG-Tailor的性能显著更优，并且可以无缝集成作为场景生成和机器人操作任务中的插件模块。
URL
https://arxiv.org/abs/2503.18988
PDF
-
HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generation
2025-03-21 04:36:57Hou In Derek Pun, Hou In Ivan Tam, Austin T. Wang, Xiaoliang Huo, Angel X. Chang, Manolis SavvaAbstract
Despite advances in indoor 3D scene layout generation, synthesizing scenes with dense object arrangements remains challenging. Existing methods primarily focus on large furniture while neglecting smaller objects, resulting in unrealistically empty scenes. Those that place small objects typically do not honor arrangement specifications, resulting in largely random placement not following the text description. We present HSM, a hierarchical framework for indoor scene generation with dense object arrangements across spatial scales. Indoor scenes are inherently hierarchical, with surfaces supporting objects at different scales, from large furniture on floors to smaller objects on tables and shelves. HSM embraces this hierarchy and exploits recurring cross-scale spatial patterns to generate complex and realistic indoor scenes in a unified manner. Our experiments show that HSM outperforms existing methods by generating scenes that are more realistic and better conform to user input across room types and spatial configurations.
Abstract (translated)
尽管在室内三维场景布局生成方面已经取得了进展，但合成包含密集物体排列的场景仍然具有挑战性。现有方法主要关注大型家具而忽视了较小的物品，导致生成的场景显得过于空旷。那些放置小物件的方法通常不遵循具体的排列规范，导致位置分布很大程度上是随机的，并不符合文本描述的要求。我们提出了HSM（分层框架），用于在不同空间尺度下生成包含密集物体布置的室内场景。室内场景本质上具有层次性，从地面上的大件家具到桌面上或架子上的小型物品，在不同的尺度上都有支持其摆放的表面。HSM采纳了这种层级结构，并利用重复出现的空间规模间的模式来统一生成复杂且真实的室内场景。实验结果表明，与现有方法相比，HSM在不同房间类型和空间配置下能生成更逼真、更能符合用户输入要求的场景。
URL
https://arxiv.org/abs/2503.16848
PDF
-
NuiScene: Exploring Efficient Generation of Unbounded Outdoor Scenes
2025-03-20 17:37:43Han-Hung Lee, Qinghong Han, Angel X. ChangAbstract
In this paper, we explore the task of generating expansive outdoor scenes, ranging from castles to high-rises. Unlike indoor scene generation, which has been a primary focus of prior work, outdoor scene generation presents unique challenges, including wide variations in scene heights and the need for a method capable of rapidly producing large landscapes. To address this, we propose an efficient approach that encodes scene chunks as uniform vector sets, offering better compression and performance than the spatially structured latents used in prior methods. Furthermore, we train an explicit outpainting model for unbounded generation, which improves coherence compared to prior resampling-based inpainting schemes while also speeding up generation by eliminating extra diffusion steps. To facilitate this task, we curate NuiScene43, a small but high-quality set of scenes, preprocessed for joint training. Notably, when trained on scenes of varying styles, our model can blend different environments, such as rural houses and city skyscrapers, within the same scene, highlighting the potential of our curation process to leverage heterogeneous scenes for joint training.
Abstract (translated)
在这篇论文中，我们探讨了生成广阔户外场景的任务，这些场景从城堡到高层建筑不等。与先前研究主要关注的室内场景生成不同，户外场景生成面临着独特的挑战，包括场景高度变化大以及需要一种能够快速生产大规模景观的方法。为了解决这些问题，我们提出了一种高效的方法，该方法将场景块编码成均匀向量集合，相比之前方法使用的空间结构化潜在变量，这种方法提供了更好的压缩和性能表现。 此外，我们还训练了一个显式的“outpainting”模型用于无边界生成，与先前基于重采样的填充方案相比，这种方案能够提高连贯性，并通过消除额外的扩散步骤加快了生成速度。为了便于这项任务，我们整理了NuiScene43，这是一个虽小但质量高的一组场景集，经过预处理以进行联合训练。 值得注意的是，在不同风格场景上训练时，我们的模型可以在同一场景中融合不同的环境，如乡村房屋和城市摩天大楼，这突显出我们的数据整理过程能够利用异构场景进行联合训练的潜力。
URL
https://arxiv.org/abs/2503.16375
PDF
-
Cube: A Roblox View of 3D Intelligence
2025-03-19 17:52:17Foundation AI Team, Kiran Bhat, Nishchaie Khanna, Karun Channa, Tinghui Zhou, Yiheng Zhu, Xiaoxia Sun, Charles Shang, Anirudh Sudarshan, Maurice Chu, Daiqing Li, Kangle Deng, Jean-Philippe Fauconnier, Tijmen Verhulsdonck, Maneesh Agrawala, Kayvon Fatahalian, Alexander Weiss, Christian Reiser, Ravi Kiran Chirravuri, Ravali Kandur, Alejandro Pelaez, Akash Garg, Michael Palleschi, Jessica Wang, Skylar Litz, Leon Liu, Anying Li, David Harmon, Derek Liu, Liangjun Feng, Denis Goupil, Lukas Kuczynski, Jihyun Yoon, Naveen Marri, Peiye Zhuang, Yinan Zhang, Brian Yin, Haomiao Jiang, Marcel van Workum, Thomas Lane, Bryce Erickson, Salil Pathare, Kyle Price, Anupam Singh, David BaszuckiAbstract
Foundation models trained on vast amounts of data have demonstrated remarkable reasoning and generation capabilities in the domains of text, images, audio and video. Our goal at Roblox is to build such a foundation model for 3D intelligence, a model that can support developers in producing all aspects of a Roblox experience, from generating 3D objects and scenes to rigging characters for animation to producing programmatic scripts describing object behaviors. We discuss three key design requirements for such a 3D foundation model and then present our first step towards building such a model. We expect that 3D geometric shapes will be a core data type and describe our solution for 3D shape tokenizer. We show how our tokenization scheme can be used in applications for text-to-shape generation, shape-to-text generation and text-to-scene generation. We demonstrate how these applications can collaborate with existing large language models (LLMs) to perform scene analysis and reasoning. We conclude with a discussion outlining our path to building a fully unified foundation model for 3D intelligence.
Abstract (translated)
基于大量数据训练的基础模型在文本、图像、音频和视频领域展示了出色的推理和生成能力。我们的目标是在Roblox上构建一个面向三维智能的基础模型，该模型能够支持开发者从生成三维物体和场景到为动画搭建角色再到编写描述对象行为的程序脚本等各个方面的工作。我们将讨论这样一个3D基础模型的关键设计要求，并介绍我们朝此方向迈出的第一步。我们预期3D几何形状将成为一种核心数据类型，并且会详细说明我们的3D形状标记化解决方案。我们会展示如何使用我们的标记化方案在文本到形状生成、形状到文本生成和文本到场景生成等应用中进行操作。此外，我们还会演示这些应用程序如何与现有的大规模语言模型（LLMs）协作以执行场景分析和推理工作。最后，我们将讨论构建一个完全统一的面向3D智能的基础模型的道路。 具体而言： 1. **核心设计要求**：为了创建能够支持开发者在Roblox上实现各种功能的3D基础模型，该模型需要具备理解、生成及修改三维数据的能力。 2. **初步步骤**：我们首先确定了3D几何形状作为核心数据类型，并开发了一个专门用于处理这些形状的标记化系统。这个3D形状标记器是构建更复杂模型的第一步。 3. **应用实例与协作**： - 文本到形状生成（Text-to-Shape Generation）和形状到文本生成（Shape-to-Text Generation）：通过使用我们的标记方案，可以将自然语言描述转换为具体的三维物体或场景，并且反过来也能从几何结构中提取出有意义的文本信息。 - 文本到场景生成（Text-to-Scene Generation）：这一功能使用户能够直接输入描述来创建复杂的三维环境，极大地简化了创作流程。 4. **与现有LLMs的合作**： 通过整合3D基础模型和现有的大规模语言模型（如GPT系列），我们可以利用LLM的文本处理能力来进行更高级别的场景分析及推理工作。这不仅能够增强用户交互体验，还为自动化内容生成开辟了新的可能。 5. **未来展望**：我们期望这些初步尝试将成为构建一个全面支持三维智能的基础模型的重要基石，在这一过程中推动更多创新和应用的发展。 综上所述，我们的目标是建立一套完整统一的3D基础模型体系，这将显著提高开发者在Roblox平台上的创造力与生产力。
URL
https://arxiv.org/abs/2503.15475
PDF
-
DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generation
2025-03-19 13:49:48Jiazhe Guo, Yikang Ding, Xiwu Chen, Shuo Chen, Bohan Li, Yingshuang Zou, Xiaoyang Lyu, Feiyang Tan, Xiaojuan Qi, Zhiheng Li, Hao ZhaoAbstract
Current generative models struggle to synthesize dynamic 4D driving scenes that simultaneously support temporal extrapolation and spatial novel view synthesis (NVS) without per-scene optimization. A key challenge lies in finding an efficient and generalizable geometric representation that seamlessly connects temporal and spatial synthesis. To address this, we propose DiST-4D, the first disentangled spatiotemporal diffusion framework for 4D driving scene generation, which leverages metric depth as the core geometric representation. DiST-4D decomposes the problem into two diffusion processes: DiST-T, which predicts future metric depth and multi-view RGB sequences directly from past observations, and DiST-S, which enables spatial NVS by training only on existing viewpoints while enforcing cycle consistency. This cycle consistency mechanism introduces a forward-backward rendering constraint, reducing the generalization gap between observed and unseen viewpoints. Metric depth is essential for both accurate reliable forecasting and accurate spatial NVS, as it provides a view-consistent geometric representation that generalizes well to unseen perspectives. Experiments demonstrate that DiST-4D achieves state-of-the-art performance in both temporal prediction and NVS tasks, while also delivering competitive performance in planning-related evaluations.
Abstract (translated)
当前的生成模型在合成动态4D驾驶场景时遇到了挑战，这些场景需要同时支持时间外推和空间新视角合成（NVS），而无需针对每个场景进行优化。关键难题在于寻找一种高效且通用化的几何表示方法，能够无缝地连接时间与空间的合成过程。为了解决这一问题，我们提出了DiST-4D，这是一种用于生成4D驾驶场景的分解时空扩散框架，它利用了度量深度作为核心几何表示。 DiST-4D将问题分解成两个扩散过程：DiST-T和DiST-S。其中，DiST-T能够直接从过去的观测数据预测未来的度量深度以及多视角RGB序列；而DiST-S则通过仅在现有视角上训练并强制执行循环一致性来实现空间NVS。这种循环一致性机制引入了前向-后向渲染约束，从而减少了观察到的和未见过的观点之间的泛化差距。 度量深度对于准确可靠的预测和精确的空间NVS都是至关重要的，因为它提供了一种视图一致性的几何表示方法，并且能够很好地推广应用于未知视角。实验结果表明，DiST-4D在时间预测任务和NVS任务中都达到了最先进的性能水平，在与规划相关的评估方面也表现出竞争性表现。
URL
https://arxiv.org/abs/2503.15208
PDF
-
SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesis
2025-03-18 22:02:35Hou In Ivan Tam, Hou In Derek Pun, Austin T. Wang, Angel X. Chang, Manolis SavvaAbstract
Despite recent advances in text-conditioned 3D indoor scene generation, there remain gaps in the evaluation of these methods. Existing metrics primarily assess the realism of generated scenes by comparing them to a set of ground-truth scenes, often overlooking alignment with the input text - a critical factor in determining how effectively a method meets user requirements. We present SceneEval, an evaluation framework designed to address this limitation. SceneEval includes metrics for both explicit user requirements, such as the presence of specific objects and their attributes described in the input text, and implicit expectations, like the absence of object collisions, providing a comprehensive assessment of scene quality. To facilitate evaluation, we introduce SceneEval-100, a dataset of scene descriptions with annotated ground-truth scene properties. We evaluate recent scene generation methods using SceneEval and demonstrate its ability to provide detailed assessments of the generated scenes, highlighting strengths and areas for improvement across multiple dimensions. Our results show that current methods struggle at generating scenes that meet user requirements, underscoring the need for further research in this direction.
Abstract (translated)
尽管在文本条件下的3D室内场景生成方面取得了进展，但这些方法的评估仍然存在不足之处。现有的度量标准主要通过将生成的场景与一组真实的场景进行比较来衡量其逼真程度，往往忽略了与输入文本的一致性——这是决定一种方法能否有效满足用户需求的关键因素。我们提出了SceneEval这一评估框架，旨在解决这一局限性。SceneEval包括了针对明确的用户要求（如输入文本中描述的具体对象及其属性）和隐含期望（例如没有物体碰撞情况）的度量标准，从而对场景质量进行全面评估。为了便于评估，我们引入了包含注释的真实场景属性的100个场景描述的数据集SceneEval-100。我们使用SceneEval来评估最近的场景生成方法，并展示了它能够提供详细的生成场景评估结果，突出显示在多个维度上的优点和改进空间。我们的结果显示，当前的方法难以生成满足用户需求的场景，强调了在此方向上进行进一步研究的需求。
URL
https://arxiv.org/abs/2503.14756
PDF
-
Advances in 4D Generation: A Survey
2025-03-18 17:59:51Qiaowei Miao, Kehan Li, Jinsheng Quan, Zhiyuan Min, Shaojie Ma, Yichao Xu, Yi Yang, Yawei LuoAbstract
Generative artificial intelligence has witnessed remarkable advancements across multiple domains in recent years. Building on the successes of 2D and 3D content generation, 4D generation, which incorporates the temporal dimension into generative tasks, has emerged as a burgeoning yet rapidly evolving research area. This paper presents a comprehensive survey of this emerging field, systematically examining its theoretical foundations, key methodologies, and practical applications, with the aim of providing readers with a holistic understanding of the current state and future potential of 4D generation. We begin by introducing the core concepts of 4D data representations, encompassing both structured and unstructured formats, and their implications for generative tasks. Building upon this foundation, we delve into the enabling technologies that drive 4D generation, including advancements in spatiotemporal modeling, neural representations, and generative frameworks. We further review recent studies that employ diverse control mechanisms and representation strategies for generating 4D outputs, categorizing these approaches and summarizing their research trajectories. In addition, we explore the wide-ranging applications of 4D generation techniques, spanning dynamic object modeling, scene generation, digital human synthesis, 4D content editing, and autonomous driving. Finally, we analyze the key challenges inherent to 4D generation, such as data availability, computational efficiency, and spatiotemporal consistency, and propose promising directions for future research. Our code is publicly available at: \href{this https URL}{this https URL}.
Abstract (translated)
近年来，生成式人工智能在多个领域取得了显著进展。继二维和三维内容生成的成功之后，四维生成作为一种新兴但迅速发展的研究领域应运而生，它将时间维度融入到生成任务中。本文旨在对这一新兴领域的现状进行全面的综述，系统地审视其理论基础、关键方法和技术应用，为读者提供关于四维生成当前状态和未来潜力的整体理解。我们首先介绍四维数据表示的核心概念，包括结构化和非结构化格式，并探讨它们在生成任务中的影响。在此基础上，我们将深入研究推动四维生成的技术，如时空建模的进步、神经网络表示以及生成框架的发展。此外，本文还回顾了最近使用多种控制机制和表示策略来生成四维输出的研究工作，对这些方法进行分类并总结其研究路径。我们进一步探讨四维生成技术的广泛应用领域，包括动态对象建模、场景生成、数字人类合成、四维内容编辑及自动驾驶等。最后，本文分析了四维生成所面临的挑战，如数据可用性、计算效率和时空一致性，并提出未来研究的潜在方向。 我们的代码可在以下链接获取：[此 HTTPS URL](this https URL)。
URL
https://arxiv.org/abs/2503.14501
PDF
-
Bolt3D: Generating 3D Scenes in Seconds
2025-03-18 17:24:19Stanislaw Szymanowicz, Jason Y. Zhang, Pratul Srinivasan, Ruiqi Gao, Arthur Brussee, Aleksander Holynski, Ricardo Martin-Brualla, Jonathan T. Barron, Philipp HenzlerAbstract
We present a latent diffusion model for fast feed-forward 3D scene generation. Given one or more images, our model Bolt3D directly samples a 3D scene representation in less than seven seconds on a single GPU. We achieve this by leveraging powerful and scalable existing 2D diffusion network architectures to produce consistent high-fidelity 3D scene representations. To train this model, we create a large-scale multiview-consistent dataset of 3D geometry and appearance by applying state-of-the-art dense 3D reconstruction techniques to existing multiview image datasets. Compared to prior multiview generative models that require per-scene optimization for 3D reconstruction, Bolt3D reduces the inference cost by a factor of up to 300 times.
Abstract (translated)
我们提出了一种用于快速前馈生成三维场景的潜在扩散模型。给定一个或多个图像，我们的模型Bolt3D可以直接在单个GPU上以不到七秒的时间采样出高质量的三维场景表示。通过利用强大的、可扩展的现有二维扩散网络架构，我们的模型能够生成一致且高保真的三维场景表示。为了训练这个模型，我们创建了一个大规模的多视角一致数据集，该数据集包含了3D几何和外观信息，这是通过对现有的多视图图像数据集应用最先进的密集三维重建技术得到的。与之前的需要针对每个场景进行三维重建优化的多视图生成模型相比，Bolt3D将推理成本降低了高达300倍。
URL
https://arxiv.org/abs/2503.14445
PDF
-
SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model
2025-03-18 06:41:02Xinqing Li, Ruiqi Song, Qingyu Xie, Ye Wu, Nanxin Zeng, Yunfeng AiAbstract
With the rapid advancement of autonomous driving technology, a lack of data has become a major obstacle to enhancing perception model accuracy. Researchers are now exploring controllable data generation using world models to diversify datasets. However, previous work has been limited to studying image generation quality on specific public datasets. There is still relatively little research on how to build data generation engines for real-world application scenes to achieve large-scale data generation for challenging scenes. In this paper, a simulator-conditioned scene generation engine based on world model is proposed. By constructing a simulation system consistent with real-world scenes, simulation data and labels, which serve as the conditions for data generation in the world model, for any scenes can be collected. It is a novel data generation pipeline by combining the powerful scene simulation capabilities of the simulation engine with the robust data generation capabilities of the world model. In addition, a benchmark with proportionally constructed virtual and real data, is provided for exploring the capabilities of world models in real-world scenes. Quantitative results show that these generated images significantly improve downstream perception models performance. Finally, we explored the generative performance of the world model in urban autonomous driving scenarios. All the data and code will be available at this https URL.
Abstract (translated)
随着自动驾驶技术的迅速发展，数据不足已成为提高感知模型精度的主要障碍。研究人员正探索使用世界模型进行可控的数据生成以丰富数据集。然而，之前的工作主要集中在特定公开数据集上的图像生成质量研究上，对于如何构建适用于真实应用场景的数据生成引擎以实现大规模挑战性场景下的数据生成的研究还相对较少。 本文提出了一种基于世界模型的仿真条件化场景生成引擎。通过构建与现实场景一致的仿真系统，可以收集任何场景所需的模拟数据和标签作为在世界模型中进行数据生成的条件。这是一种创新性的数据生成流程，它结合了仿真引擎强大的场景模拟能力和世界模型的数据生成能力。 此外，本文还提供了一个由虚拟数据和实际数据按比例构建的基准测试平台，用以探索世界模型在现实场景中的能力。定量结果表明，这些生成的图像显著提升了下游感知模型的表现性能。最后，我们研究了世界模型在城市自动驾驶场景下的生成性能。所有数据及代码将在该网址提供：[此 URL]
URL
https://arxiv.org/abs/2503.13952
PDF
