Scene_Generation
-
Bolt3D: Generating 3D Scenes in Seconds
2025-03-18 17:24:19Stanislaw Szymanowicz, Jason Y. Zhang, Pratul Srinivasan, Ruiqi Gao, Arthur Brussee, Aleksander Holynski, Ricardo Martin-Brualla, Jonathan T. Barron, Philipp HenzlerAbstract
We present a latent diffusion model for fast feed-forward 3D scene generation. Given one or more images, our model Bolt3D directly samples a 3D scene representation in less than seven seconds on a single GPU. We achieve this by leveraging powerful and scalable existing 2D diffusion network architectures to produce consistent high-fidelity 3D scene representations. To train this model, we create a large-scale multiview-consistent dataset of 3D geometry and appearance by applying state-of-the-art dense 3D reconstruction techniques to existing multiview image datasets. Compared to prior multiview generative models that require per-scene optimization for 3D reconstruction, Bolt3D reduces the inference cost by a factor of up to 300 times.
Abstract (translated)
我们提出了一种用于快速前馈生成三维场景的潜在扩散模型。给定一个或多个图像，我们的模型Bolt3D可以直接在单个GPU上以不到七秒的时间采样出高质量的三维场景表示。通过利用强大的、可扩展的现有二维扩散网络架构，我们的模型能够生成一致且高保真的三维场景表示。为了训练这个模型，我们创建了一个大规模的多视角一致数据集，该数据集包含了3D几何和外观信息，这是通过对现有的多视图图像数据集应用最先进的密集三维重建技术得到的。与之前的需要针对每个场景进行三维重建优化的多视图生成模型相比，Bolt3D将推理成本降低了高达300倍。
URL
https://arxiv.org/abs/2503.14445
PDF
-
SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model
2025-03-18 06:41:02Xinqing Li, Ruiqi Song, Qingyu Xie, Ye Wu, Nanxin Zeng, Yunfeng AiAbstract
With the rapid advancement of autonomous driving technology, a lack of data has become a major obstacle to enhancing perception model accuracy. Researchers are now exploring controllable data generation using world models to diversify datasets. However, previous work has been limited to studying image generation quality on specific public datasets. There is still relatively little research on how to build data generation engines for real-world application scenes to achieve large-scale data generation for challenging scenes. In this paper, a simulator-conditioned scene generation engine based on world model is proposed. By constructing a simulation system consistent with real-world scenes, simulation data and labels, which serve as the conditions for data generation in the world model, for any scenes can be collected. It is a novel data generation pipeline by combining the powerful scene simulation capabilities of the simulation engine with the robust data generation capabilities of the world model. In addition, a benchmark with proportionally constructed virtual and real data, is provided for exploring the capabilities of world models in real-world scenes. Quantitative results show that these generated images significantly improve downstream perception models performance. Finally, we explored the generative performance of the world model in urban autonomous driving scenarios. All the data and code will be available at this https URL.
Abstract (translated)
随着自动驾驶技术的迅速发展，数据不足已成为提高感知模型精度的主要障碍。研究人员正探索使用世界模型进行可控的数据生成以丰富数据集。然而，之前的工作主要集中在特定公开数据集上的图像生成质量研究上，对于如何构建适用于真实应用场景的数据生成引擎以实现大规模挑战性场景下的数据生成的研究还相对较少。 本文提出了一种基于世界模型的仿真条件化场景生成引擎。通过构建与现实场景一致的仿真系统，可以收集任何场景所需的模拟数据和标签作为在世界模型中进行数据生成的条件。这是一种创新性的数据生成流程，它结合了仿真引擎强大的场景模拟能力和世界模型的数据生成能力。 此外，本文还提供了一个由虚拟数据和实际数据按比例构建的基准测试平台，用以探索世界模型在现实场景中的能力。定量结果表明，这些生成的图像显著提升了下游感知模型的表现性能。最后，我们研究了世界模型在城市自动驾驶场景下的生成性能。所有数据及代码将在该网址提供：[此 URL]
URL
https://arxiv.org/abs/2503.13952
PDF
-
ChatBEV: A Visual Language Model that Understands BEV Maps
2025-03-18 06:12:38Qingyao Xu, Siheng Chen, Guang Chen, Yanfeng Wang, Ya ZhangAbstract
Traffic scene understanding is essential for intelligent transportation systems and autonomous driving, ensuring safe and efficient vehicle operation. While recent advancements in VLMs have shown promise for holistic scene understanding, the application of VLMs to traffic scenarios, particularly using BEV maps, remains under explored. Existing methods often suffer from limited task design and narrow data amount, hindering comprehensive scene understanding. To address these challenges, we introduce ChatBEV-QA, a novel BEV VQA benchmark contains over 137k questions, designed to encompass a wide range of scene understanding tasks, including global scene understanding, vehicle-lane interactions, and vehicle-vehicle interactions. This benchmark is constructed using an novel data collection pipeline that generates scalable and informative VQA data for BEV maps. We further fine-tune a specialized vision-language model ChatBEV, enabling it to interpret diverse question prompts and extract relevant context-aware information from BEV maps. Additionally, we propose a language-driven traffic scene generation pipeline, where ChatBEV facilitates map understanding and text-aligned navigation guidance, significantly enhancing the generation of realistic and consistent traffic scenarios. The dataset, code and the fine-tuned model will be released.
Abstract (translated)
交通场景理解对于智能运输系统和自动驾驶至关重要，确保车辆的安全高效运行。虽然最近视觉语言模型（VLMs）在全面场景理解方面取得了显著进展，但将这些技术应用于基于鸟瞰图（BEV）的地图的交通场景中仍处于探索阶段。现有方法通常受限于任务设计狭窄及数据量有限的问题，这阻碍了对复杂交通场景的理解。为了解决这些问题，我们引入了一种新的基准测试ChatBEV-QA，这是一个包含超过137,000个问题的BEV视觉问答（VQA）基准测试，旨在涵盖广泛的场景理解任务，包括全局场景理解、车辆与车道之间的互动以及车辆间的互动。该基准测试通过一个创新的数据收集流程构建而成，能够生成用于BEV地图的大规模和信息丰富的VQA数据。 我们进一步微调了一个专门的视觉语言模型ChatBEV，使其能够解析各种问题提示，并从BEV地图中提取相关且上下文相关的资讯。此外，我们还提出了一种由语言驱动的交通场景生成流程，在此过程中，ChatBEV促进了对地图的理解以及文本与导航指导的一致性，显著提升了现实主义和一致性的交通场景生成。 该数据集、代码及微调后的模型将在未来发布。
URL
https://arxiv.org/abs/2503.13938
PDF
-
Seeing the Future, Perceiving the Future: A Unified Driving World Model for Future Generation and Perception
2025-03-17 17:59:50Dingkang Liang, Dingyuan Zhang, Xin Zhou, Sifan Tu, Tianrui Feng, Xiaofan Li, Yumeng Zhang, Mingyang Du, Xiao Tan, Xiang BaiAbstract
We present UniFuture, a simple yet effective driving world model that seamlessly integrates future scene generation and perception within a single framework. Unlike existing models focusing solely on pixel-level future prediction or geometric reasoning, our approach jointly models future appearance (i.e., RGB image) and geometry (i.e., depth), ensuring coherent predictions. Specifically, during the training, we first introduce a Dual-Latent Sharing scheme, which transfers image and depth sequence in a shared latent space, allowing both modalities to benefit from shared feature learning. Additionally, we propose a Multi-scale Latent Interaction mechanism, which facilitates bidirectional refinement between image and depth features at multiple spatial scales, effectively enhancing geometry consistency and perceptual alignment. During testing, our UniFuture can easily predict high-consistency future image-depth pairs by only using the current image as input. Extensive experiments on the nuScenes dataset demonstrate that UniFuture outperforms specialized models on future generation and perception tasks, highlighting the advantages of a unified, structurally-aware world model. The project page is at this https URL.
Abstract (translated)
我们介绍了UniFuture，这是一种简单而有效的驾驶世界模型，它在一个单一框架内无缝整合了未来场景生成和感知。与现有的仅专注于像素级未来预测或几何推理的模型不同，我们的方法同时建模未来的外观（即RGB图像）和几何结构（即深度），确保一致的预测结果。具体来说，在训练阶段，我们首先引入了一种双潜在共享方案，该方案在一个共享的潜在空间中传输图像和深度序列，使两种模式都能从共同特征学习中获益。此外，我们还提出了一种多尺度潜互动机制，这一机制促进了在多个空间尺度下图象与深度特征之间的双向细化过程，有效增强了几何一致性及感知对齐。 测试阶段，UniFuture可以通过仅使用当前图像作为输入来轻易预测高一致性的未来图像-深度配对。在nuScenes数据集上的广泛实验表明，UniFuture在未来的生成和感知任务中超越了专门化的模型，突显出统一的、结构意识强的世界模型的优势。项目的网页链接如下：[此URL]（请将方括号中的文字替换为实际网址）。
URL
https://arxiv.org/abs/2503.13587
PDF
-
SteerX: Creating Any Camera-Free 3D and 4D Scenes with Geometric Steering
2025-03-15 07:20:08Byeongjun Park, Hyojun Go, Hyelin Nam, Byung-Hoon Kim, Hyungjin Chung, Changick KimAbstract
Recent progress in 3D/4D scene generation emphasizes the importance of physical alignment throughout video generation and scene reconstruction. However, existing methods improve the alignment separately at each stage, making it difficult to manage subtle misalignments arising from another stage. Here, we present SteerX, a zero-shot inference-time steering method that unifies scene reconstruction into the generation process, tilting data distributions toward better geometric alignment. To this end, we introduce two geometric reward functions for 3D/4D scene generation by using pose-free feed-forward scene reconstruction models. Through extensive experiments, we demonstrate the effectiveness of SteerX in improving 3D/4D scene generation.
Abstract (translated)
最近在3D和4D场景生成方面的进展强调了在整个视频生成和场景重建过程中物理对齐的重要性。然而，现有的方法是在每个阶段单独改进对齐，这使得难以管理由其他阶段产生的微妙不对齐问题。在这里，我们介绍了SteerX，这是一种零样本推理时间引导方法，它将场景重建统一到生成过程之中，使数据分布向更好的几何对齐方向倾斜。为此，我们通过使用无姿态的前馈场景重构模型引入了两个用于3D和4D场景生成的几何奖励函数。通过广泛的实验，我们展示了SteerX在改进3D和4D场景生成方面的有效性。
URL
https://arxiv.org/abs/2503.12024
PDF
-
WonderVerse: Extendable 3D Scene Generation with Video Generative Models
2025-03-12 08:44:51Hao Feng, Zhi Zuo, Jia-hui Pan, Ka-hei Hui, Yi-hua Shao, Qi Dou, Wei Xie, Zheng-zhe LiuAbstract
We introduce \textit{WonderVerse}, a simple but effective framework for generating extendable 3D scenes. Unlike existing methods that rely on iterative depth estimation and image inpainting, often leading to geometric distortions and inconsistencies, WonderVerse leverages the powerful world-level priors embedded within video generative foundation models to create highly immersive and geometrically coherent 3D environments. Furthermore, we propose a new technique for controllable 3D scene extension to substantially increase the scale of the generated environments. Besides, we introduce a novel abnormal sequence detection module that utilizes camera trajectory to address geometric inconsistency in the generated videos. Finally, WonderVerse is compatible with various 3D reconstruction methods, allowing both efficient and high-quality generation. Extensive experiments on 3D scene generation demonstrate that our WonderVerse, with an elegant and simple pipeline, delivers extendable and highly-realistic 3D scenes, markedly outperforming existing works that rely on more complex architectures.
Abstract (translated)
我们介绍了一个名为\textit{WonderVerse}的框架，这是一个简单却有效的生成可扩展三维场景的方法。与现有的依赖于迭代深度估计和图像修补方法不同，后者常常导致几何扭曲和不一致性，WonderVerse 利用了视频生成基础模型中嵌入的世界级先验知识来创建高度沉浸式且几何一致的3D环境。此外，我们提出了一种可控的3D场景扩展技术，以显著增加生成环境的规模。另外，我们引入了一个新的异常序列检测模块，该模块利用摄像机轨迹解决生成视频中的几何不一致性问题。最后，WonderVerse 兼容各种3D重建方法，既支持高效又高质量的生成。 在三维场景生成方面的广泛实验表明，我们的 WonderVerse 通过一个优雅且简单的管道，能够提供可扩展和高度逼真的3D场景，显著优于依赖更复杂架构的现有工作。
URL
https://arxiv.org/abs/2503.09160
PDF
-
Controllable 3D Outdoor Scene Generation via Scene Graphs
2025-03-10 10:26:08Yuheng Liu, Xinke Li, Yuning Zhang, Lu Qi, Xin Li, Wenping Wang, Chongshou Li, Xueting Li, Ming-Hsuan YangAbstract
Three-dimensional scene generation is crucial in computer vision, with applications spanning autonomous driving, gaming and the metaverse. Current methods either lack user control or rely on imprecise, non-intuitive conditions. In this work, we propose a method that uses, scene graphs, an accessible, user friendly control format to generate outdoor 3D scenes. We develop an interactive system that transforms a sparse scene graph into a dense BEV (Bird's Eye View) Embedding Map, which guides a conditional diffusion model to generate 3D scenes that match the scene graph description. During inference, users can easily create or modify scene graphs to generate large-scale outdoor scenes. We create a large-scale dataset with paired scene graphs and 3D semantic scenes to train the BEV embedding and diffusion models. Experimental results show that our approach consistently produces high-quality 3D urban scenes closely aligned with the input scene graphs. To the best of our knowledge, this is the first approach to generate 3D outdoor scenes conditioned on scene graphs.
Abstract (translated)
三维场景生成在计算机视觉领域至关重要，其应用范围涵盖自动驾驶、游戏以及元宇宙。当前的方法要么缺乏用户控制能力，要么依赖于不精确且不易理解的条件。在这项工作中，我们提出了一种使用场景图（一种易用且友好的用户控制格式）来生成户外3D场景的方法。我们开发了一个交互式系统，该系统将稀疏的场景图转换为密集的BEV（鸟瞰视图）嵌入地图，并指导有条件的扩散模型生成与场景图描述相匹配的3D场景。在推理过程中，用户可以轻松创建或修改场景图以生成大规模户外场景。我们创建了一个大型数据集，其中包括配对的场景图和三维语义场景，用于训练BEV嵌入和扩散模型。实验结果表明，我们的方法始终能够产生高质量、与输入场景图紧密一致的3D城市场景。据我们所知，这是第一个基于场景图生成三维户外场景的方法。
URL
https://arxiv.org/abs/2503.07152
PDF
-
Unlocking Generalization for Robotics via Modularity and Scale
Abstract
How can we build generalist robot systems? Scale may not be enough due to the significant multimodality of robotics tasks, lack of easily accessible data and the challenges of deploying on physical hardware. Meanwhile, most deployed robotic systems today are inherently modular and can leverage the independent generalization capabilities of each module to perform well. Therefore, this thesis seeks to tackle the task of building generalist robot agents by integrating these components into one: combining modularity with large-scale learning for general purpose robot control. The first question we consider is: how can we build modularity and hierarchy into learning systems? Our key insight is that rather than having the agent learn hierarchy and low-level control end-to-end, we can enforce modularity via planning to enable more efficient and capable robot learners. Next, we come to the role of scale in building generalist robot systems. To scale, neural networks require vast amounts of diverse data, expressive architectures to fit the data and a source of supervision to generate the data. We leverage a powerful supervision source: classical planning, which can generalize, but is expensive to run and requires access to privileged information to perform well in practice. We use these planners to supervise large-scale policy learning in simulation to produce generalist agents. Finally, we consider how to unify modularity with large-scale policy learning to build real-world robot systems capable of performing zero-shot manipulation. We do so by tightly integrating key ingredients of modular high and mid-level planning, learned local control, procedural scene generation and large-scale policy learning for sim2real transfer. We demonstrate that this recipe can produce a single, generalist agent that can solve challenging long-horizon manipulation tasks in the real world.
Abstract (translated)
如何构建通用机器人系统？由于机器人任务的多模态特性、缺乏易于获取的数据以及在物理硬件上部署所面临的挑战，单纯依赖规模可能并不足够。然而，当今大多数已部署的机器人系统本质上是模块化的，并能够利用每个模块独立泛化的能力来实现高性能。因此，本文旨在通过将这些组件整合为一个整体来解决构建通用型机器人代理的问题：结合模块化与大规模学习以用于通用目的机器人的控制。 首先，我们考虑的是如何在学习系统中引入模块化和层次结构？我们的关键见解是，与其让代理人从头到尾（端到端）学会层次结构和低级控制，不如通过规划来强制执行模块化，从而实现更高效且更具能力的机器人学习者。接下来，我们要讨论规模在构建通用型机器人系统中的作用。为了达到大规模，神经网络需要大量的多样化数据、能够适应这些数据的表达性架构以及生成数据的监督源。我们利用了一个强大的监督来源：经典规划，它可以泛化，但在实际运行中代价高昂，并且需要访问特权信息才能表现良好。我们使用这些规划器在模拟环境中监督大规模策略学习，以产生通用型代理。 最后，我们要考虑如何将模块化与大规模策略学习统一起来，以构建能够在零样本操作中执行的现实世界机器人系统。为此，我们将关键成分紧密结合：模块化的高级和中级计划、学习本地控制、程序化场景生成以及用于模拟到真实环境转换的大规模策略学习。我们证明了这一配方可以产生一个单一的、通用型代理，能够解决实际环境中具有挑战性的长时间操作任务。 总的来说，本文通过结合模块化设计与大规模学习的方法来探索构建能够在多种情况下执行复杂任务的一般性机器人系统的新途径。
URL
https://arxiv.org/abs/2503.06814
PDF
-
FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement
2025-03-06 19:34:15Ian Huang, Yanan Bao, Karen Truong, Howard Zhou, Cordelia Schmid, Leonidas Guibas, Alireza FathiAbstract
Scene generation with 3D assets presents a complex challenge, requiring both high-level semantic understanding and low-level geometric reasoning. While Multimodal Large Language Models (MLLMs) excel at semantic tasks, their application to 3D scene generation is hindered by their limited grounding on 3D geometry. In this paper, we investigate how to best work with MLLMs in an object placement task. Towards this goal, we introduce a novel framework, FirePlace, that applies existing MLLMs in (1) 3D geometric reasoning and the extraction of relevant geometric details from the 3D scene, (2) constructing and solving geometric constraints on the extracted low-level geometry, and (3) pruning for final placements that conform to common sense. By combining geometric reasoning with real-world understanding of MLLMs, our method can propose object placements that satisfy both geometric constraints as well as high-level semantic common-sense considerations. Our experiments show that these capabilities allow our method to place objects more effectively in complex scenes with intricate geometry, surpassing the quality of prior work.
Abstract (translated)
使用3D资产生成场景是一项复杂的挑战，需要同时具备高层次的语义理解和低层次的几何推理能力。虽然多模态大型语言模型（MLLMs）在语义任务上表现出色，但它们在应用于3D场景生成时受到对3D几何理解有限的影响。在这篇论文中，我们探讨了如何最有效地将MLLMs应用到物体放置的任务中。为此，我们引入了一个新颖的框架FirePlace，该框架利用现有的MLLMs进行： 1. 3D几何推理，并从3D场景中提取相关的几何细节； 2. 构建并解决基于提取出的低层次几何的约束条件； 3. 剪枝以获得符合常识的最终放置位置。 通过结合几何推理和MLLMs对现实世界的理解，我们的方法可以提出既满足几何约束又能考虑高层次语义常识的对象放置建议。实验表明，这些能力使我们的方法能够更有效地在具有复杂几何结构的场景中放置物体，并且其质量超过了先前的工作。
URL
https://arxiv.org/abs/2503.04919
PDF
-
DualDiff+: Dual-Branch Diffusion for High-Fidelity Video Generation with Reward Guidance
2025-03-05 17:31:45Zhao Yang, Zezhong Qian, Xiaofan Li, Weixiang Xu, Gongpeng Zhao, Ruohong Yu, Lingsi Zhu, Longjun LiuAbstract
Accurate and high-fidelity driving scene reconstruction demands the effective utilization of comprehensive scene information as conditional inputs. Existing methods predominantly rely on 3D bounding boxes and BEV road maps for foreground and background control, which fail to capture the full complexity of driving scenes and adequately integrate multimodal information. In this work, we present DualDiff, a dual-branch conditional diffusion model designed to enhance driving scene generation across multiple views and video sequences. Specifically, we introduce Occupancy Ray-shape Sampling (ORS) as a conditional input, offering rich foreground and background semantics alongside 3D spatial geometry to precisely control the generation of both elements. To improve the synthesis of fine-grained foreground objects, particularly complex and distant ones, we propose a Foreground-Aware Mask (FGM) denoising loss function. Additionally, we develop the Semantic Fusion Attention (SFA) mechanism to dynamically prioritize relevant information and suppress noise, enabling more effective multimodal fusion. Finally, to ensure high-quality image-to-video generation, we introduce the Reward-Guided Diffusion (RGD) framework, which maintains global consistency and semantic coherence in generated videos. Extensive experiments demonstrate that DualDiff achieves state-of-the-art (SOTA) performance across multiple datasets. On the NuScenes dataset, DualDiff reduces the FID score by 4.09% compared to the best baseline. In downstream tasks, such as BEV segmentation, our method improves vehicle mIoU by 4.50% and road mIoU by 1.70%, while in BEV 3D object detection, the foreground mAP increases by 1.46%. Code will be made available at this https URL.
Abstract (translated)
准确且高保真的驾驶场景重建需要有效地利用全面的场景信息作为条件输入。现有的方法主要依赖于三维边界框和鸟瞰图道路地图来控制前景和背景，这无法捕捉驾驶场景的全部复杂性，并且不足以充分融合多模态信息。在本工作中，我们提出了DualDiff，这是一种双分支条件扩散模型，旨在增强从多个视角和视频序列生成驾驶场景的能力。具体来说，我们引入了Occupancy Ray-shape Sampling（ORS）作为条件输入，提供丰富的前景和背景语义以及三维空间几何信息来精确控制两者的生成。 为了改进细粒度前景对象的合成，特别是复杂且距离较远的对象，我们提出了一种基于前景的掩码去噪损失函数（FGM）。此外，我们开发了语义融合注意机制（SFA），以动态优先处理相关信息并抑制噪声，从而实现更有效的多模态融合。 最后，为了确保高质量的图像到视频生成，我们引入了引导式扩散框架（RGD），它在生成的视频中保持全局一致性和语义连贯性。广泛的实验表明，DualDiff在多个数据集上达到了最先进的性能。在NuScenes数据集上，与最佳基线相比，DualDiff将FID得分降低了4.09%。在下游任务（如鸟瞰图分割）中，我们的方法使车辆的mIoU提高了4.50%，道路的mIoU提高了1.70%，而在鸟瞰图3D对象检测中，前景的mAP增加了1.46%。 代码将在[此链接](https://this-is-the-url-to-be-filled-in-later.com)提供。
URL
https://arxiv.org/abs/2503.03689
PDF
-
A Survey on Text-Driven 360-Degree Panorama Generation
2025-02-20 18:19:57Hai Wang, Xiaoyu Xiang, Weihao Xia, Jing-Hao XueAbstract
The advent of text-driven 360-degree panorama generation, enabling the synthesis of 360-degree panoramic images directly from textual descriptions, marks a transformative advancement in immersive visual content creation. This innovation significantly simplifies the traditionally complex process of producing such content. Recent progress in text-to-image diffusion models has accelerated the rapid development in this emerging field. This survey presents a comprehensive review of text-driven 360-degree panorama generation, offering an in-depth analysis of state-of-the-art algorithms and their expanding applications in 360-degree 3D scene generation. Furthermore, we critically examine current limitations and propose promising directions for future research. A curated project page with relevant resources and research papers is available at this https URL.
Abstract (translated)
文本驱动的360度全景图像生成技术的出现，使得可以直接从文字描述中合成360度全景图，标志着沉浸式视觉内容创作领域的革命性进步。这一创新大大简化了传统上复杂的内容生产过程。最近在文本到图像扩散模型方面取得的进步加速了该新兴领域的发展。本综述对文本驱动的360度全景生成进行了全面回顾，并深入分析了当前最先进的算法及其在360度三维场景生成中的扩展应用。此外，我们还批判性地审视了现有局限性并提出了未来研究的有前景的方向。相关资源和研究论文可在以下网址（此 https URL）上查阅。
URL
https://arxiv.org/abs/2502.14799
PDF
-
Rolling Ahead Diffusion for Traffic Scene Simulation
2025-02-13 18:45:56Yunpeng Liu, Matthew Niedoba, William Harvey, Adam Scibior, Berend Zwartsenberg, Frank WoodAbstract
Realistic driving simulation requires that NPCs not only mimic natural driving behaviors but also react to the behavior of other simulated agents. Recent developments in diffusion-based scenario generation focus on creating diverse and realistic traffic scenarios by jointly modelling the motion of all the agents in the scene. However, these traffic scenarios do not react when the motion of agents deviates from their modelled trajectories. For example, the ego-agent can be controlled by a stand along motion planner. To produce reactive scenarios with joint scenario models, the model must regenerate the scenario at each timestep based on new observations in a Model Predictive Control (MPC) fashion. Although reactive, this method is time-consuming, as one complete possible future for all NPCs is generated per simulation step. Alternatively, one can utilize an autoregressive model (AR) to predict only the immediate next-step future for all NPCs. Although faster, this method lacks the capability for advanced planning. We present a rolling diffusion based traffic scene generation model which mixes the benefits of both methods by predicting the next step future and simultaneously predicting partially noised further future steps at the same time. We show that such model is efficient compared to diffusion model based AR, achieving a beneficial compromise between reactivity and computational efficiency.
Abstract (translated)
现实中的驾驶模拟要求NPC不仅模仿自然的驾驶行为，还能够对其他仿真代理的行为作出反应。近期基于扩散模型的情景生成技术着重于通过共同建模场景中所有代理的运动来创建多样且逼真的交通情景。然而，这些交通情景在代理的运动偏离其预设轨迹时并不会做出反应。例如，主控代理（ego-agent）可以由独立的路径规划器控制。为了利用联合情景模型生成响应式的情景，模型必须基于新的观察结果以模型预测控制（MPC）的方式，在每个时间步重新生成场景。尽管这种方法具备响应性，但它非常耗时，因为每模拟一步就需要为所有NPC生成一个完整的可能未来。 另一种方法是使用自回归模型（AR），仅预测所有NPC的下一步未来的运动。虽然此法较快，但缺乏对未来进行深入规划的能力。我们提出了一种滚动扩散基交通场景生成模型，该模型结合了上述两种方法的优点：它同时预测接下来一步的情景和部分模糊化后的未来步骤。我们展示了这种模型相比于基于扩散模型的自回归模型在效率上具有优势，并且能够在响应性和计算效率之间取得有利的平衡。 这种方法通过混合预测方式，即预测下一步的具体行为同时对更远期的行为做出概率性推测，从而实现了更高的灵活性和效率，在保持场景真实性的前提下提高了系统的响应速度。
URL
https://arxiv.org/abs/2502.09587
PDF
-
MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generation
2025-02-09 12:23:40Zhifei Yang, Keyang Lu, Chao Zhang, Jiaxing Qi, Hanqi Jiang, Ruifei Ma, Shenglin Yin, Yifan Xu, Mingzhe Xing, Zhen Xiao, Jieyi Long, Xiangde Liu, Guangyao ZhaiAbstract
Controllable 3D scene generation has extensive applications in virtual reality and interior design, where the generated scenes should exhibit high levels of realism and controllability in terms of geometry. Scene graphs provide a suitable data representation that facilitates these applications. However, current graph-based methods for scene generation are constrained to text-based inputs and exhibit insufficient adaptability to flexible user inputs, hindering the ability to precisely control object geometry. To address this issue, we propose MMGDreamer, a dual-branch diffusion model for scene generation that incorporates a novel Mixed-Modality Graph, visual enhancement module, and relation predictor. The mixed-modality graph allows object nodes to integrate textual and visual modalities, with optional relationships between nodes. It enhances adaptability to flexible user inputs and enables meticulous control over the geometry of objects in the generated scenes. The visual enhancement module enriches the visual fidelity of text-only nodes by constructing visual representations using text embeddings. Furthermore, our relation predictor leverages node representations to infer absent relationships between nodes, resulting in more coherent scene layouts. Extensive experimental results demonstrate that MMGDreamer exhibits superior control of object geometry, achieving state-of-the-art scene generation performance. Project page: this https URL.
Abstract (translated)
可控的3D场景生成在虚拟现实和室内设计等领域有着广泛的应用，生成的场景需要表现出高度的真实感与几何形状上的可控制性。场景图提供了一种合适的数据表示方式，便于实现这些应用需求。然而，当前基于图形的方法进行场景生成时，受限于文本输入，并且对于灵活多变的用户输入适应性不足，这限制了精确控制物体几何结构的能力。 为了解决这个问题，我们提出了MMGDreamer，这是一种用于场景生成的双分支扩散模型，集成了新颖的混合模态图、视觉增强模块和关系预测器。这种混合模态图允许对象节点整合文本和视觉两种模式，并支持节点间的可选关系连接。这提升了对灵活用户输入的适应性，并使得精确控制生成场景中物体的几何形状成为可能。 此外，MMGDreamer中的视觉增强模块通过使用文本嵌入来构建视觉表示，从而丰富了仅基于文本节点的视觉保真度。我们的关系预测器利用节点表示来推断节点间缺失的关系，进而产生更为连贯的场景布局。 广泛的实验结果表明，MMGDreamer在物体几何控制方面表现出色，并达到了目前最先进的场景生成性能水平。项目页面链接：[此URL](this https URL)（请替换为实际链接）。
URL
https://arxiv.org/abs/2502.05874
PDF
-
LAYOUTDREAMER: Physics-guided Layout for Text-to-3D Compositional Scene Generation
2025-02-04 02:51:37Yang Zhou, Zongjin He, Qixuan Li, Chao WangAbstract
Recently, the field of text-guided 3D scene generation has garnered significant attention. High-quality generation that aligns with physical realism and high controllability is crucial for practical 3D scene applications. However, existing methods face fundamental limitations: (i) difficulty capturing complex relationships between multiple objects described in the text, (ii) inability to generate physically plausible scene layouts, and (iii) lack of controllability and extensibility in compositional scenes. In this paper, we introduce LayoutDreamer, a framework that leverages 3D Gaussian Splatting (3DGS) to facilitate high-quality, physically consistent compositional scene generation guided by text. Specifically, given a text prompt, we convert it into a directed scene graph and adaptively adjust the density and layout of the initial compositional 3D Gaussians. Subsequently, dynamic camera adjustments are made based on the training focal point to ensure entity-level generation quality. Finally, by extracting directed dependencies from the scene graph, we tailor physical and layout energy to ensure both realism and flexibility. Comprehensive experiments demonstrate that LayoutDreamer outperforms other compositional scene generation quality and semantic alignment methods. Specifically, it achieves state-of-the-art (SOTA) performance in the multiple objects generation metric of T3Bench.
Abstract (translated)
最近，文本引导的三维场景生成领域引起了广泛关注。高质量地生成符合物理真实性和高度可控性的三维场景对于实际应用至关重要。然而，现有的方法面临一些根本性的限制：（i）难以捕捉文本描述中多个对象之间的复杂关系；（ii）无法生成合乎物理逻辑的场景布局；（iii）在组合式场景中的可控制性与扩展性不足。为此，在本文中我们介绍了LayoutDreamer框架，该框架利用3D高斯点阵(3DGS)技术来实现高质量、物理上一致且由文本引导的组合式三维场景生成。具体而言，给定一个文本提示后，我们将它转换为有向场景图，并根据初始的组合式3D高斯分布自适应地调整其密度和布局。随后，基于训练焦点进行动态相机调整以确保实体级别的生成质量。最后，通过从场景图中提取定向依赖关系，我们定制物理能量与布局能量，以此保证现实性和灵活性。全面的实验表明，LayoutDreamer在组合式三维场景生成质量和语义对齐方法方面优于其他模型，并且在T3Bench的多对象生成度量标准上达到了最新的技术水平（SOTA）。
URL
https://arxiv.org/abs/2502.01949
PDF
-
PhiP-G: Physics-Guided Text-to-3D Compositional Scene Generation
2025-02-02 07:47:03Qixuan Li, Chao Wang, Zongjin He, Yan PengAbstract
Text-to-3D asset generation has achieved significant optimization under the supervision of 2D diffusion priors. However, when dealing with compositional scenes, existing methods encounter several challenges: 1). failure to ensure that composite scene layouts comply with physical laws; 2). difficulty in accurately capturing the assets and relationships described in complex scene descriptions; 3). limited autonomous asset generation capabilities among layout approaches leveraging large language models (LLMs). To avoid these compromises, we propose a novel framework for compositional scene generation, PhiP-G, which seamlessly integrates generation techniques with layout guidance based on a world model. Leveraging LLM-based agents, PhiP-G analyzes the complex scene description to generate a scene graph, and integrating a multimodal 2D generation agent and a 3D Gaussian generation method for targeted assets creation. For the stage of layout, PhiP-G employs a physical pool with adhesion capabilities and a visual supervision agent, forming a world model for layout prediction and planning. Extensive experiments demonstrate that PhiP-G significantly enhances the generation quality and physical rationality of the compositional scenes. Notably, PhiP-G attains state-of-the-art (SOTA) performance in CLIP scores, achieves parity with the leading methods in generation quality as measured by the T$^3$Bench, and improves efficiency by 24x.
Abstract (translated)
在二维扩散先验的监督下，文本到三维资产生成已经实现了显著优化。然而，在处理组合场景时，现有方法面临以下挑战：1) 无法确保复合场景布局符合物理规律；2) 难以准确捕捉复杂场景描述中的资产和关系；3) 利用大型语言模型（LLMs）的布局方法在自主资产生成能力方面存在局限。为避免这些妥协，我们提出了一种新颖的框架PhiP-G，该框架无缝地将生成技术与基于世界模型的布局指导相结合。 通过利用基于大语言模型的代理，PhiP-G 分析复杂场景描述来生成场景图，并整合多模态二维生成代理和三维高斯生成方法以创建特定资产。在布局阶段，PhiP-G 采用具有粘合能力的物理池和视觉监督代理，形成一个世界模型用于布局预测和规划。 广泛的实验表明，PhiP-G 显著提升了组合场景生成的质量和物理合理性。特别地，PhiP-G 在CLIP评分中达到最先进的（SOTA）性能，在T$^3$Bench 测量的生成质量上与领先方法持平，并且在效率方面提高了24倍。
URL
https://arxiv.org/abs/2502.00708
PDF
-
HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation
2025-01-24 18:59:51Xin Zhou, Dingkang Liang, Sifan Tu, Xiwu Chen, Yikang Ding, Dingyuan Zhang, Feiyang Tan, Hengshuang Zhao, Xiang BaiAbstract
Driving World Models (DWMs) have become essential for autonomous driving by enabling future scene prediction. However, existing DWMs are limited to scene generation and fail to incorporate scene understanding, which involves interpreting and reasoning about the driving environment. In this paper, we present a unified Driving World Model named HERMES. We seamlessly integrate 3D scene understanding and future scene evolution (generation) through a unified framework in driving scenarios. Specifically, HERMES leverages a Bird's-Eye View (BEV) representation to consolidate multi-view spatial information while preserving geometric relationships and interactions. We also introduce world queries, which incorporate world knowledge into BEV features via causal attention in the Large Language Model (LLM), enabling contextual enrichment for understanding and generation tasks. We conduct comprehensive studies on nuScenes and OmniDrive-nuScenes datasets to validate the effectiveness of our method. HERMES achieves state-of-the-art performance, reducing generation error by 32.4% and improving understanding metrics such as CIDEr by 8.0%. The model and code will be publicly released at this https URL.
Abstract (translated)
驾驶世界模型（DWMs）已成为自动驾驶不可或缺的一部分，通过实现未来场景预测来支持自动驾驶。然而，现有的DWM仅限于生成新场景，并未能整合对驾驶环境的理解，即无法解读和推理驾驶环境中的情况。在本文中，我们提出了一种统一的驾驶世界模型HERMES。我们在驾驶场景下通过一个统一框架无缝地集成了三维场景理解和未来场景演变（生成）。具体而言，HERMES利用鸟瞰图（BEV）表示来整合多视角空间信息的同时保持几何关系和互动性。此外，我们还引入了世界查询机制，该机制通过在大型语言模型中应用因果注意力将世界知识融入到BEV特征中，从而为理解和生成任务提供上下文增强。我们在nuScenes和OmniDrive-nuScenes数据集上进行了全面的研究以验证我们的方法的有效性。HERMES取得了最先进的性能，在减少生成误差方面比现有技术降低了32.4%，在理解指标如CIDEr的评估中提高了8.0%。该模型及其代码将在[此链接](https://this https URL)公开发布。
URL
https://arxiv.org/abs/2501.14729
PDF
-
CityDreamer4D: Compositional Generative Model of Unbounded 4D Cities
2025-01-15 17:59:56Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, Ziwei LiuAbstract
3D scene generation has garnered growing attention in recent years and has made significant progress. Generating 4D cities is more challenging than 3D scenes due to the presence of structurally complex, visually diverse objects like buildings and vehicles, and heightened human sensitivity to distortions in urban environments. To tackle these issues, we propose CityDreamer4D, a compositional generative model specifically tailored for generating unbounded 4D cities. Our main insights are 1) 4D city generation should separate dynamic objects (e.g., vehicles) from static scenes (e.g., buildings and roads), and 2) all objects in the 4D scene should be composed of different types of neural fields for buildings, vehicles, and background stuff. Specifically, we propose Traffic Scenario Generator and Unbounded Layout Generator to produce dynamic traffic scenarios and static city layouts using a highly compact BEV representation. Objects in 4D cities are generated by combining stuff-oriented and instance-oriented neural fields for background stuff, buildings, and vehicles. To suit the distinct characteristics of background stuff and instances, the neural fields employ customized generative hash grids and periodic positional embeddings as scene parameterizations. Furthermore, we offer a comprehensive suite of datasets for city generation, including OSM, GoogleEarth, and CityTopia. The OSM dataset provides a variety of real-world city layouts, while the Google Earth and CityTopia datasets deliver large-scale, high-quality city imagery complete with 3D instance annotations. Leveraging its compositional design, CityDreamer4D supports a range of downstream applications, such as instance editing, city stylization, and urban simulation, while delivering state-of-the-art performance in generating realistic 4D cities.
Abstract (translated)
近年来，三维场景生成受到了越来越多的关注，并取得了显著进展。然而，生成四维城市（包含时间维度的变化）比单纯生成三维场景更具挑战性，这是因为建筑物和车辆等结构复杂、视觉多样化的物体的存在以及人们对城市环境中任何失真的高度敏感性。为了解决这些问题，我们提出了CityDreamer4D，这是一种专门用于生成无边界四维城市的组合式生成模型。 我们的主要见解是： 1. 四维城市的生成应该将动态对象（例如车辆）与静态场景（例如建筑物和道路）区分开来。 2. 在四维场景中的所有物体都应由不同类型神经场组成，包括针对建筑物、车辆及背景物质的神经场。 具体来说，我们提出了交通场景生成器和无边界布局生成器，这两个生成器使用高度紧凑的BEV（鸟瞰图）表示方式来产生动态交通场景和静态城市布局。在四维城市中，物体通过结合面向物质和实例化的神经场（背景物质、建筑物和车辆）进行生成。 为了适应背景物质和实例的不同特性，这些神经场采用了定制化生成哈希网格和周期位置嵌入作为场景参数化方式。 此外，我们提供了一整套用于城市生成的数据集，包括OSM、Google Earth 和 CityTopia。其中OSM数据集提供了多种真实世界的城市布局，而 Google 地球和CityTopia 数据集则提供了大规模、高质量的城市图像，并附有3D实例标注信息。借助其组合式设计，CityDreamer4D 支持一系列下游应用，包括实例编辑、城市风格化及都市仿真等，同时在生成逼真的四维城市方面表现出色。
URL
https://arxiv.org/abs/2501.08983
PDF
-
BloomScene: Lightweight Structured 3D Gaussian Splatting for Crossmodal Scene Generation
2025-01-15 11:33:34Xiaolu Hou, Mingcheng Li, Dingkang Yang, Jiawei Chen, Ziyun Qian, Xiao Zhao, Yue Jiang, Jinjie Wei, Qingyao Xu, Lihua ZhangAbstract
With the widespread use of virtual reality applications, 3D scene generation has become a new challenging research frontier. 3D scenes have highly complex structures and need to ensure that the output is dense, coherent, and contains all necessary structures. Many current 3D scene generation methods rely on pre-trained text-to-image diffusion models and monocular depth estimators. However, the generated scenes occupy large amounts of storage space and often lack effective regularisation methods, leading to geometric distortions. To this end, we propose BloomScene, a lightweight structured 3D Gaussian splatting for crossmodal scene generation, which creates diverse and high-quality 3D scenes from text or image inputs. Specifically, a crossmodal progressive scene generation framework is proposed to generate coherent scenes utilizing incremental point cloud reconstruction and 3D Gaussian splatting. Additionally, we propose a hierarchical depth prior-based regularization mechanism that utilizes multi-level constraints on depth accuracy and smoothness to enhance the realism and continuity of the generated scenes. Ultimately, we propose a structured context-guided compression mechanism that exploits structured hash grids to model the context of unorganized anchor attributes, which significantly eliminates structural redundancy and reduces storage overhead. Comprehensive experiments across multiple scenes demonstrate the significant potential and advantages of our framework compared with several baselines.
Abstract (translated)
随着虚拟现实应用的广泛使用，3D场景生成已经成为一个新的研究前沿。3D场景具有高度复杂的结构，并且需要确保输出是密集、连贯并且包含所有必要的结构。许多当前的3D场景生成方法依赖于预训练的文字到图像扩散模型和单目深度估计器。然而，这些方法产生的场景占用大量的存储空间，并且往往缺乏有效的正则化方法，导致几何变形。为了解决这些问题，我们提出了BloomScene，这是一种轻量级的结构化的3D高斯点着色方法，用于跨模态场景生成，可以从文本或图像输入创建多样性和高质量的3D场景。 具体而言，我们提出了一种跨模式逐步场景生成框架，利用增量点云重建和3D高斯点着色来生成连贯的场景。此外，我们还提出了基于分层深度先验的正则化机制，通过在深度准确度和平滑度上应用多层次约束，增强生成场景的真实感和连续性。最终，我们提出了一种结构化的上下文引导压缩机制，利用有组织的哈希网格来建模无序锚点属性的上下文信息，从而显著消除结构冗余并减少存储开销。 在多个场景上的综合实验表明，我们的框架与几种基准方法相比具有重要的潜力和优势。
URL
https://arxiv.org/abs/2501.10462
PDF
-
StarGen: A Spatiotemporal Autoregression Framework with Video Diffusion Model for Scalable and Controllable Scene Generation
2025-01-10 07:41:47Shangjin Zhai, Zhichao Ye, Jialin Liu, Weijian Xie, Jiaqi Hu, Zhen Peng, Hua Xue, Danpeng Chen, Xiaomeng Wang, Lei Yang, Nan Wang, Haomin Liu, Guofeng ZhangAbstract
Recent advances in large reconstruction and generative models have significantly improved scene reconstruction and novel view generation. However, due to compute limitations, each inference with these large models is confined to a small area, making long-range consistent scene generation challenging. To address this, we propose StarGen, a novel framework that employs a pre-trained video diffusion model in an autoregressive manner for long-range scene generation. The generation of each video clip is conditioned on the 3D warping of spatially adjacent images and the temporally overlapping image from previously generated clips, improving spatiotemporal consistency in long-range scene generation with precise pose control. The spatiotemporal condition is compatible with various input conditions, facilitating diverse tasks, including sparse view interpolation, perpetual view generation, and layout-conditioned city generation. Quantitative and qualitative evaluations demonstrate StarGen's superior scalability, fidelity, and pose accuracy compared to state-of-the-art methods.
Abstract (translated)
最近在大规模重建和生成模型方面的进展显著提升了场景重建和新颖视图生成的性能。然而，由于计算限制，这些大型模型每次推理只能处理一个小区域，这使得长距离的一致性场景生成变得具有挑战性。为了解决这个问题，我们提出了StarGen，这是一个新的框架，它采用预训练的视频扩散模型以自回归方式用于长距离场景生成。每个视频片段的生成受到空间相邻图像和之前生成片段中时间上重叠的图像的空间扭曲条件的影响，这提高了长时间范围内场景生成中的时空一致性，并实现了精确的姿态控制。这种时空条件可以与各种输入条件兼容，从而能够支持多种任务，包括稀疏视图插值、持续视图生成以及布局条件下的城市生成。定量和定性的评估表明，StarGen在可扩展性、保真度和姿态准确性方面优于现有的方法。
URL
https://arxiv.org/abs/2501.05763
PDF
-
Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priors
2025-01-05 12:20:13Minglin Chen, Longguang Wang, Sheng Ao, Ye Zhang, Kai Xu, Yulan GuoAbstract
3D scene generation conditioned on text prompts has significantly progressed due to the development of 2D diffusion generation models. However, the textual description of 3D scenes is inherently inaccurate and lacks fine-grained control during training, leading to implausible scene generation. As an intuitive and feasible solution, the 3D layout allows for precise specification of object locations within the scene. To this end, we present a text-to-scene generation method (namely, Layout2Scene) using additional semantic layout as the prompt to inject precise control of 3D object positions. Specifically, we first introduce a scene hybrid representation to decouple objects and backgrounds, which is initialized via a pre-trained text-to-3D model. Then, we propose a two-stage scheme to optimize the geometry and appearance of the initialized scene separately. To fully leverage 2D diffusion priors in geometry and appearance generation, we introduce a semantic-guided geometry diffusion model and a semantic-geometry guided diffusion model which are finetuned on a scene dataset. Extensive experiments demonstrate that our method can generate more plausible and realistic scenes as compared to state-of-the-art approaches. Furthermore, the generated scene allows for flexible yet precise editing, thereby facilitating multiple downstream applications.
Abstract (translated)
基于文本提示的3D场景生成由于2D扩散生成模型的发展而取得了显著进步。然而，对3D场景的文字描述本质上不够准确，并且在训练过程中缺乏细粒度控制，导致生成的场景不合理。作为一种直观可行的解决方案，3D布局允许精确指定场景中对象的位置。为此，我们提出了一种使用附加语义布局作为提示来注入对3D物体位置精准控制的文本到场景生成方法（即Layout2Scene）。具体而言，我们首先引入一种场景混合表示法，以分离物体和背景，并通过预训练的文本到3D模型进行初始化。然后，我们提出了一个两阶段方案，分别优化初始化场景的几何形状和外观。为了充分利用在几何形状和外观生成中的2D扩散先验知识，我们引入了语义引导的几何扩散模型以及基于语义-几何的引导扩散模型，并在这两类模型上使用场景数据集进行微调。 广泛的实验表明，我们的方法能够比最先进的方法生成更合理、更逼真的场景。此外，所生成的场景允许灵活而精确地编辑，从而促进了多种下游应用的发展。
URL
https://arxiv.org/abs/2501.02519
PDF
