Scene_Generation
-
ZeroScene: A Zero-Shot Framework for 3D Scene Generation from a Single Image and Controllable Texture Editing
2025-09-28 03:21:12Xiang Tang, Ruotong Li, Xiaopeng FanAbstract
In the field of 3D content generation, single image scene reconstruction methods still struggle to simultaneously ensure the quality of individual assets and the coherence of the overall scene in complex environments, while texture editing techniques often fail to maintain both local continuity and multi-view consistency. In this paper, we propose a novel system ZeroScene, which leverages the prior knowledge of large vision models to accomplish both single image-to-3D scene reconstruction and texture editing in a zero-shot manner. ZeroScene extracts object-level 2D segmentation and depth information from input images to infer spatial relationships within the scene. It then jointly optimizes 3D and 2D projection losses of the point cloud to update object poses for precise scene alignment, ultimately constructing a coherent and complete 3D scene that encompasses both foreground and background. Moreover, ZeroScene supports texture editing of objects in the scene. By imposing constraints on the diffusion model and introducing a mask-guided progressive image generation strategy, we effectively maintain texture consistency across multiple viewpoints and further enhance the realism of rendered results through Physically Based Rendering (PBR) material estimation. Experimental results demonstrate that our framework not only ensures the geometric and appearance accuracy of generated assets, but also faithfully reconstructs scene layouts and produces highly detailed textures that closely align with text prompts.
Abstract (translated)
在3D内容生成领域，单张图像场景重建方法仍难以同时保证复杂环境中各个资产的质量和整个场景的连贯性。此外，纹理编辑技术往往无法兼顾局部连续性和多视角一致性。为此，本文提出了一种新的系统ZeroScene，该系统利用大型视觉模型的先验知识，以零样本的方式实现单张图像到3D场景重建及纹理编辑。 ZeroScene从输入图像中提取对象级别的2D分割和深度信息，以此推断出场景中的空间关系。随后，通过联合优化点云的3D与2D投影损失来更新物体姿态，从而精确地对齐场景，并最终构建一个包含前景与背景且连贯完整的3D场景。 此外，ZeroScene还支持场景中对象的纹理编辑。通过对扩散模型施加约束并引入基于蒙版引导的渐进式图像生成策略，我们能够有效保持多视角下的纹理一致性，并通过物理基础渲染（PBR）材质估计进一步提升渲染结果的真实感。 实验结果显示，我们的框架不仅保证了生成资产的几何和外观准确性，还忠实还原场景布局，并产生与文本提示高度一致且细节丰富的纹理。
URL
https://arxiv.org/abs/2509.23607
PDF
-
WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving
2025-09-27 16:47:44Ziyue Zhu, Zhanqian Wu, Zhenxin Zhu, Lijun Zhou, Haiyang Sun, Bing Wan, Kun Ma, Guang Chen, Hangjun Ye, Jin Xie, jian YangAbstract
Recent advances in driving-scene generation and reconstruction have demonstrated significant potential for enhancing autonomous driving systems by producing scalable and controllable training data. Existing generation methods primarily focus on synthesizing diverse and high-fidelity driving videos; however, due to limited 3D consistency and sparse viewpoint coverage, they struggle to support convenient and high-quality novel-view synthesis (NVS). Conversely, recent 3D/4D reconstruction approaches have significantly improved NVS for real-world driving scenes, yet inherently lack generative capabilities. To overcome this dilemma between scene generation and reconstruction, we propose \textbf{WorldSplat}, a novel feed-forward framework for 4D driving-scene generation. Our approach effectively generates consistent multi-track videos through two key steps: ((i)) We introduce a 4D-aware latent diffusion model integrating multi-modal information to produce pixel-aligned 4D Gaussians in a feed-forward manner. ((ii)) Subsequently, we refine the novel view videos rendered from these Gaussians using a enhanced video diffusion model. Extensive experiments conducted on benchmark datasets demonstrate that \textbf{WorldSplat} effectively generates high-fidelity, temporally and spatially consistent multi-track novel view driving videos.
Abstract (translated)
最近在驾驶场景生成和重建方面的进展展示了其增强自动驾驶系统的能力，通过生产可扩展且可控的训练数据。现有的生成方法主要集中在合成多样性和高保真的驾驶视频上；然而，由于三维一致性有限和视角覆盖稀疏的问题，它们难以支持便捷、高质量的新视点合成（NVS）。相比之下，最近的3D/4D重建方法在真实世界的驾驶场景中显著提高了NVS的质量，但天生缺乏生成能力。为了克服这种在场景生成与重建之间的困境，我们提出了**WorldSplat**，这是一种新颖的前馈框架，用于生成四维驾驶场景。我们的方法通过两个关键步骤有效地产生了具有一致性的多轨道视频：（i）我们引入了一种4D感知的潜在扩散模型，该模型集成了多模态信息，并以前馈方式生成像素对齐的4D高斯分布。（ii）随后，利用增强的视频扩散模型来细化从这些高斯分布渲染出的新视点视频。在基准数据集上进行的一系列实验表明，**WorldSplat**能够有效生成高度保真、时间和空间一致性的多轨道新视点驾驶视频。
URL
https://arxiv.org/abs/2509.23402
PDF
-
MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning
2025-09-26 12:46:00Jinkun Hao, Naifu Liang, Zhen Luo, Xudong Xu, Weipeng Zhong, Ran Yi, Yichen Jin, Zhaoyang Lyu, Feng Zheng, Lizhuang Ma, Jiangmiao PangAbstract
The ability of robots to interpret human instructions and execute manipulation tasks necessitates the availability of task-relevant tabletop scenes for training. However, traditional methods for creating these scenes rely on time-consuming manual layout design or purely randomized layouts, which are limited in terms of plausibility or alignment with the tasks. In this paper, we formulate a novel task, namely task-oriented tabletop scene generation, which poses significant challenges due to the substantial gap between high-level task instructions and the tabletop scenes. To support research on such a challenging task, we introduce MesaTask-10K, a large-scale dataset comprising approximately 10,700 synthetic tabletop scenes with manually crafted layouts that ensure realistic layouts and intricate inter-object relations. To bridge the gap between tasks and scenes, we propose a Spatial Reasoning Chain that decomposes the generation process into object inference, spatial interrelation reasoning, and scene graph construction for the final 3D layout. We present MesaTask, an LLM-based framework that utilizes this reasoning chain and is further enhanced with DPO algorithms to generate physically plausible tabletop scenes that align well with given task descriptions. Exhaustive experiments demonstrate the superior performance of MesaTask compared to baselines in generating task-conforming tabletop scenes with realistic layouts. Project page is at this https URL
Abstract (translated)
机器人解释人类指令并执行操作任务的能力需要具有与训练任务相关的桌面场景的可用性。然而，传统的创建这些场景的方法依赖于耗时的手动布局设计或完全随机化的布局设计，这两种方法在合理性和与任务的相关性方面都存在局限。 本文提出了一项新任务——目标导向型桌面场景生成（task-oriented tabletop scene generation），这项任务由于高层次的任务指令与实际的桌面场景之间的巨大差距而面临重大挑战。为了支持对此类具有挑战性的任务的研究，我们引入了MesaTask-10K数据集，这是一个大规模的数据集，包含大约10,700个合成桌面场景，这些场景的手动布局设计确保了现实主义和复杂的对象间关系。 为弥合任务与场景之间的差距，我们提出了一个空间推理链（Spatial Reasoning Chain），该链条将生成过程分解为物体推断、空间相互关联推理以及最终的3D布局场景图构建。我们提出了一种基于大型语言模型（LLM）的框架——MesaTask，该框架利用了上述推理链，并通过加入DPO算法进一步增强了其功能，以生成与给定任务描述相符且物理上合理的桌面场景。 详尽的实验表明，在生成符合任务要求、布局真实感强的桌面场景方面，MesaTask相较于基准方法具有显著的优势。项目页面位于[此处](https://该项目的网址在这里)。
URL
https://arxiv.org/abs/2509.22281
PDF
-
Drag4D: Align Your Motion with Text-Driven 3D Scene Generation
2025-09-26 05:23:45Minjun Kang, Inkyu Shin, Taeyeop Lee, In So Kweon, Kuk-Jin YoonAbstract
We introduce Drag4D, an interactive framework that integrates object motion control within text-driven 3D scene generation. This framework enables users to define 3D trajectories for the 3D objects generated from a single image, seamlessly integrating them into a high-quality 3D background. Our Drag4D pipeline consists of three stages. First, we enhance text-to-3D background generation by applying 2D Gaussian Splatting with panoramic images and inpainted novel views, resulting in dense and visually complete 3D reconstructions. In the second stage, given a reference image of the target object, we introduce a 3D copy-and-paste approach: the target instance is extracted in a full 3D mesh using an off-the-shelf image-to-3D model and seamlessly composited into the generated 3D scene. The object mesh is then positioned within the 3D scene via our physics-aware object position learning, ensuring precise spatial alignment. Lastly, the spatially aligned object is temporally animated along a user-defined 3D trajectory. To mitigate motion hallucination and ensure view-consistent temporal alignment, we develop a part-augmented, motion-conditioned video diffusion model that processes multiview image pairs together with their projected 2D trajectories. We demonstrate the effectiveness of our unified architecture through evaluations at each stage and in the final results, showcasing the harmonized alignment of user-controlled object motion within a high-quality 3D background.
Abstract (translated)
我们介绍了一种名为Drag4D的交互式框架，该框架集成了基于文本驱动的三维场景生成中的物体运动控制。此框架允许用户为从单张图像生成的三维物体定义三维轨迹，并将它们无缝地融入高质量的三维背景中。 Drag4D的工作流程分为三个阶段： 1. **增强型文本到三维背景生成**：我们利用2D高斯光斑技术结合全景图和修复后的新型视角，增强了由文本驱动生成三维场景的能力。这使得生成的三维重建更加密集且视觉上完整。 2. **基于参考图像的目标物体3D复制与粘贴方法**：给定目标物体的参考图像后，在这一阶段中，我们将该实例从全3D网格模型中提取出来，并将其无缝地合并到生成的三维场景中。我们采用现成的图像到三维模型转换技术来实现这一点。接下来，通过我们的物理感知对象位置学习方法，将物体网格放置在三维场景中的正确位置上，确保了空间对齐的准确性。 3. **时间动画处理**：最后一步是根据用户定义的三维轨迹，在时间维度上动态调整已经精确对齐的对象。为了减少运动错觉并保证多视角的一致性，我们开发了一种部分增强、以运动条件为基础的视频扩散模型。该模型能同时处理多视角图像对及其投影后的二维轨迹。 通过在每个阶段以及最终结果中的评估，我们展示了用户控制物体运动与高质量三维背景之间的和谐统一，证明了我们统一架构的有效性和优越性。
URL
https://arxiv.org/abs/2509.21888
PDF
-
4D Driving Scene Generation With Stereo Forcing
2025-09-24 15:37:17Hao Lu, Zhuang Ma, Guangfeng Jiang, Wenhang Ge, Bohan Li, Yuzhan Cai, Wenzhao Zheng, Yunpeng Zhang, Yingcong ChenAbstract
Current generative models struggle to synthesize dynamic 4D driving scenes that simultaneously support temporal extrapolation and spatial novel view synthesis (NVS) without per-scene optimization. Bridging generation and novel view synthesis remains a major challenge. We present PhiGenesis, a unified framework for 4D scene generation that extends video generation techniques with geometric and temporal consistency. Given multi-view image sequences and camera parameters, PhiGenesis produces temporally continuous 4D Gaussian splatting representations along target 3D trajectories. In its first stage, PhiGenesis leverages a pre-trained video VAE with a novel range-view adapter to enable feed-forward 4D reconstruction from multi-view images. This architecture supports single-frame or video inputs and outputs complete 4D scenes including geometry, semantics, and motion. In the second stage, PhiGenesis introduces a geometric-guided video diffusion model, using rendered historical 4D scenes as priors to generate future views conditioned on trajectories. To address geometric exposure bias in novel views, we propose Stereo Forcing, a novel conditioning strategy that integrates geometric uncertainty during denoising. This method enhances temporal coherence by dynamically adjusting generative influence based on uncertainty-aware perturbations. Our experimental results demonstrate that our method achieves state-of-the-art performance in both appearance and geometric reconstruction, temporal generation and novel view synthesis (NVS) tasks, while simultaneously delivering competitive performance in downstream evaluations. Homepage is at \href{this https URL}{PhiGensis}.
Abstract (translated)
当前的生成模型在同时支持时间外推和空间新视图合成（NVS）的情况下，难以生成具有动态4D驾驶场景的数据。连接生成与新视图合成仍然是一个主要挑战。我们提出了PhiGenesis，这是一个用于4D场景生成的统一框架，它通过几何和时间一致性扩展了视频生成技术。给定多视角图像序列和摄像机参数后，PhiGenesis能够沿着目标3D轨迹产生连续的时间4D高斯散射表示。 在第一个阶段中，PhiGenesis利用带有新颖范围视图适配器的预训练视频VAE（变分自编码器），使从多视角图像进行前向传播式的4D重建成为可能。这种架构支持单帧或视频输入，并输出包括几何、语义和运动在内的完整4D场景。 在第二个阶段，PhiGenesis引入了一个基于几何引导的视频扩散模型，使用渲染的历史4D场景作为先验条件来生成未来的视图，这些未来视图根据轨迹进行调节。为了解决新视角中几何暴露偏差的问题，我们提出了Stereo Forcing，这是一种新的条件策略，在去噪过程中整合了几何不确定性。这种技术通过基于不确定性的扰动动态调整生成影响来增强时间一致性。 实验结果表明，我们的方法在外观和几何重建、时间生成和新视图合成（NVS）任务中均达到了最先进的性能，并且同时在下游评估中提供了竞争性表现。 PhiGenesis的主页位于[此处](https://this https URL "PhiGensis")。
URL
https://arxiv.org/abs/2509.20251
PDF
-
SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent
2025-09-24 09:06:41Yandan Yang, Baoxiong Jia, Shujie Zhang, Siyuan HuangAbstract
Indoor scene synthesis has become increasingly important with the rise of Embodied AI, which requires 3D environments that are not only visually realistic but also physically plausible and functionally diverse. While recent approaches have advanced visual fidelity, they often remain constrained to fixed scene categories, lack sufficient object-level detail and physical consistency, and struggle to align with complex user instructions. In this work, we present SceneWeaver, a reflective agentic framework that unifies diverse scene synthesis paradigms through tool-based iterative refinement. At its core, SceneWeaver employs a language model-based planner to select from a suite of extensible scene generation tools, ranging from data-driven generative models to visual- and LLM-based methods, guided by self-evaluation of physical plausibility, visual realism, and semantic alignment with user input. This closed-loop reason-act-reflect design enables the agent to identify semantic inconsistencies, invoke targeted tools, and update the environment over successive iterations. Extensive experiments on both common and open-vocabulary room types demonstrate that SceneWeaver not only outperforms prior methods on physical, visual, and semantic metrics, but also generalizes effectively to complex scenes with diverse instructions, marking a step toward general-purpose 3D environment generation. Project website: this https URL.
Abstract (translated)
随着嵌入式人工智能（Embodied AI）的发展，室内场景合成变得越来越重要。这不仅需要视觉上逼真的三维环境，还需要物理上合理且功能多样的空间。尽管最近的方法在提升视觉保真度方面取得了进展，但它们通常仍局限于固定场景类别，缺乏足够的物体级细节和物理一致性，并且难以适应复杂的用户指令。 在这项工作中，我们介绍了SceneWeaver，这是一个基于代理的框架，通过工具驱动的迭代优化统一了各种场景合成范式。SceneWeaver的核心是一个语言模型驱动的计划器，该计划器从一套可扩展的场景生成工具中进行选择，这些工具有数据驱动的生成模型、视觉方法和大型语言模型（LLM）方法等。计划过程通过自我评估物理合理性、视觉逼真度和与用户输入的语义一致性来进行指导。 这种闭环式的思考-行动-反思设计使代理能够识别语义不一致，调用特定工具，并在连续迭代中更新环境。在常见房间类型和开放式词汇表房间类型的广泛实验中，我们发现SceneWeaver不仅在物理、视觉和语义指标上优于先前的方法，而且还能有效地推广到具有多样指令的复杂场景，标志着向着通用3D环境生成迈出了一步。 项目网站：[此链接](https://this-url.com/)（请将“this https URL”替换为实际网址）。
URL
https://arxiv.org/abs/2509.20414
PDF
-
Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation
2025-09-23 17:58:01Sherwin Bahmani, Tianchang Shen, Jiawei Ren, Jiahui Huang, Yifeng Jiang, Haithem Turki, Andrea Tagliasacchi, David B. Lindell, Zan Gojcic, Sanja Fidler, Huan Ling, Jun Gao, Xuanchi RenAbstract
The ability to generate virtual environments is crucial for applications ranging from gaming to physical AI domains such as robotics, autonomous driving, and industrial AI. Current learning-based 3D reconstruction methods rely on the availability of captured real-world multi-view data, which is not always readily available. Recent advancements in video diffusion models have shown remarkable imagination capabilities, yet their 2D nature limits the applications to simulation where a robot needs to navigate and interact with the environment. In this paper, we propose a self-distillation framework that aims to distill the implicit 3D knowledge in the video diffusion models into an explicit 3D Gaussian Splatting (3DGS) representation, eliminating the need for multi-view training data. Specifically, we augment the typical RGB decoder with a 3DGS decoder, which is supervised by the output of the RGB decoder. In this approach, the 3DGS decoder can be purely trained with synthetic data generated by video diffusion models. At inference time, our model can synthesize 3D scenes from either a text prompt or a single image for real-time rendering. Our framework further extends to dynamic 3D scene generation from a monocular input video. Experimental results show that our framework achieves state-of-the-art performance in static and dynamic 3D scene generation.
Abstract (translated)
生成虚拟环境的能力对于从游戏到物理AI领域（如机器人、自动驾驶和工业AI）的应用至关重要。目前基于学习的三维重建方法依赖于捕捉到的真实世界多视角数据，而这种数据并不总是易于获取。最近视频扩散模型的进步展示了令人印象深刻的想象能力，但其二维特性限制了在需要机器人导航与环境互动等模拟场景中的应用。在这篇论文中，我们提出了一种自我蒸馏框架，旨在将视频扩散模型中的隐含三维知识转化为显式的三维高斯点（3DGS）表示形式，从而无需多视角训练数据。具体来说，我们在典型的RGB解码器上添加了一个3DGS解码器，并通过RGB解码器的输出对其进行监督。在此方法中，3DGS解码器可以完全依赖于视频扩散模型生成的合成数据进行训练。在推理时，我们的模型可以从文本提示或单张图像实时渲染三维场景。此外，该框架还可以扩展到从单一视角输入视频生成动态三维场景。实验结果表明，我们提出的框架在静态和动态三维场景生成方面达到了最先进的性能。
URL
https://arxiv.org/abs/2509.19296
PDF
-
Octree Latent Diffusion for Semantic 3D Scene Generation and Completion
2025-09-20 00:53:13Xujia Zhang, Brendan Crowe, Christoffer HeckmanAbstract
The completion, extension, and generation of 3D semantic scenes are an interrelated set of capabilities that are useful for robotic navigation and exploration. Existing approaches seek to decouple these problems and solve them oneoff. Additionally, these approaches are often domain-specific, requiring separate models for different data distributions, e.g. indoor vs. outdoor scenes. To unify these techniques and provide cross-domain compatibility, we develop a single framework that can perform scene completion, extension, and generation in both indoor and outdoor scenes, which we term Octree Latent Semantic Diffusion. Our approach operates directly on an efficient dual octree graph latent representation: a hierarchical, sparse, and memory-efficient occupancy structure. This technique disentangles synthesis into two stages: (i) structure diffusion, which predicts binary split signals to construct a coarse occupancy octree, and (ii) latent semantic diffusion, which generates semantic embeddings decoded by a graph VAE into voxellevel semantic labels. To perform semantic scene completion or extension, our model leverages inference-time latent inpainting, or outpainting respectively. These inference-time methods use partial LiDAR scans or maps to condition generation, without the need for retraining or finetuning. We demonstrate highquality structure, coherent semantics, and robust completion from single LiDAR scans, as well as zero-shot generalization to out-of-distribution LiDAR data. These results indicate that completion-through-generation in a dual octree graph latent space is a practical and scalable alternative to regression-based pipelines for real-world robotic perception tasks.
Abstract (translated)
三维语义场景的完成、扩展和生成是一系列相互关联的能力，这些能力对于机器人导航与探索非常有用。现有方法试图将这些问题解耦，并单独解决每一个问题。此外，这些方法往往是特定领域的，需要为不同的数据分布（如室内场景与室外场景）分别构建模型。为了统一这些技术并提供跨域兼容性，我们开发了一个单一的框架，能够在室内和室外场景中执行场景完成、扩展和生成任务，我们将该框架命名为Octree Latent Semantic Diffusion。 我们的方法直接在一种高效且稀疏的记忆节约型占用结构——双八叉树图隐式表示上操作。这种技术将合成过程分为两个阶段：(i) 结构扩散，预测二进制分割信号以构建一个粗略的占用八叉树；(ii) 隐含语义扩散，生成由图VAE解码成体素级别语义标签的语义嵌入。 为了执行语义场景完成或扩展任务，我们的模型利用了推断时的隐式插值（完成）和外推（扩展）。这些推断方法使用部分LiDAR扫描或地图作为条件进行生成，而无需重新训练或微调。我们展示了从单次LiDAR扫描中获得高质量结构、连贯语义以及稳健完成能力的能力，并且还能够零样本泛化到分布外的LiDAR数据。 这些结果表明，在双八叉树图隐式空间中的通过生成进行场景完成是一种实用而可扩展的方法，可以作为基于回归的工作流管道在现实世界的机器人感知任务中的一种替代方案。
URL
https://arxiv.org/abs/2509.16483
PDF
-
SPATIALGEN: Layout-guided 3D Indoor Scene Generation
2025-09-18 14:12:32Chuan Fang, Heng Li, Yixun Liang, Jia Zheng, Yongsen Mao, Yuan Liu, Rui Tang, Zihan Zhou, Ping TanAbstract
Creating high-fidelity 3D models of indoor environments is essential for applications in design, virtual reality, and robotics. However, manual 3D modeling remains time-consuming and labor-intensive. While recent advances in generative AI have enabled automated scene synthesis, existing methods often face challenges in balancing visual quality, diversity, semantic consistency, and user control. A major bottleneck is the lack of a large-scale, high-quality dataset tailored to this task. To address this gap, we introduce a comprehensive synthetic dataset, featuring 12,328 structured annotated scenes with 57,440 rooms, and 4.7M photorealistic 2D renderings. Leveraging this dataset, we present SpatialGen, a novel multi-view multi-modal diffusion model that generates realistic and semantically consistent 3D indoor scenes. Given a 3D layout and a reference image (derived from a text prompt), our model synthesizes appearance (color image), geometry (scene coordinate map), and semantic (semantic segmentation map) from arbitrary viewpoints, while preserving spatial consistency across modalities. SpatialGen consistently generates superior results to previous methods in our experiments. We are open-sourcing our data and models to empower the community and advance the field of indoor scene understanding and generation.
Abstract (translated)
创建高保真度的室内环境3D模型对于设计、虚拟现实和机器人技术的应用至关重要。然而，手动进行3D建模仍然耗时且劳动密集型。虽然最近在生成式AI方面的进展已经使得场景自动化合成成为可能，但现有的方法往往面临着平衡视觉质量、多样性、语义一致性以及用户控制的挑战。一个重要瓶颈在于缺少一个大规模高质量的数据集来专门应对这一任务。为解决这一缺口，我们介绍了一个全面的人工数据集，该数据集中包含了12,328个结构化标注场景和57,440个房间，并有4.7M张逼真的2D渲染图像。利用这一数据集，我们提出了一种新颖的多视角多模态扩散模型——SpatialGen，能够生成视觉上真实且语义一致的室内三维场景。给定一个3D布局和一张参考图片（由文本提示派生），我们的模型可以从任意视角合成立体感外观（颜色图像）、几何结构（场景坐标图）以及语义信息（语义分割图），同时在不同模态间保持空间一致性。在实验中，SpatialGen始终比先前的方法产生更好的结果。我们计划开放源代码并公开数据和模型以赋能社区，并推动室内场景理解和生成领域的发展。
URL
https://arxiv.org/abs/2509.14981
PDF
-
Causal Reasoning Elicits Controllable 3D Scene Generation
2025-09-18 01:03:21Shen Chen, Ruiyu Zhao, Jiale Zhou, Zongkai Wu, Jenq-Neng Hwang, Lei LiAbstract
Existing 3D scene generation methods often struggle to model the complex logical dependencies and physical constraints between objects, limiting their ability to adapt to dynamic and realistic environments. We propose CausalStruct, a novel framework that embeds causal reasoning into 3D scene generation. Utilizing large language models (LLMs), We construct causal graphs where nodes represent objects and attributes, while edges encode causal dependencies and physical constraints. CausalStruct iteratively refines the scene layout by enforcing causal order to determine the placement order of objects and applies causal intervention to adjust the spatial configuration according to physics-driven constraints, ensuring consistency with textual descriptions and real-world dynamics. The refined scene causal graph informs subsequent optimization steps, employing a Proportional-Integral-Derivative(PID) controller to iteratively tune object scales and positions. Our method uses text or images to guide object placement and layout in 3D scenes, with 3D Gaussian Splatting and Score Distillation Sampling improving shape accuracy and rendering stability. Extensive experiments show that CausalStruct generates 3D scenes with enhanced logical coherence, realistic spatial interactions, and robust adaptability.
Abstract (translated)
现有的三维场景生成方法通常难以模拟物体之间的复杂逻辑依赖和物理约束，这限制了它们适应动态和现实环境的能力。我们提出了一种名为CausalStruct的新框架，该框架将因果推理嵌入到3D场景生成中。通过使用大型语言模型（LLMs），我们构建了因果图，其中节点代表对象及其属性，而边则编码因果依赖性和物理约束。CausalStruct 通过强制执行因果顺序来迭代地精炼场景布局以确定物体的放置顺序，并采用因果干预方法根据物理学驱动的约束调整空间配置，确保与文本描述和现实世界动态的一致性。精炼后的场景因果图指导后续优化步骤，利用比例-积分-微分（PID）控制器迭代调优对象的比例和位置。我们的方法使用文本或图像来引导3D场景中的物体放置和布局，并通过3D高斯点投影和评分蒸馏采样提高形状精度和渲染稳定性。广泛的实验表明，CausalStruct 能够生成具有增强逻辑连贯性、逼真空间交互以及强大适应性的三维场景。
URL
https://arxiv.org/abs/2509.15249
PDF
-
A Synthetic Data Pipeline for Supporting Manufacturing SMEs in Visual Assembly Control
2025-09-16 13:48:55Jonas Werheid, Shengjie He, Aymen Gannouni, Anas Abdelrazeq, Robert H. SchmittAbstract
Quality control of assembly processes is essential in manufacturing to ensure not only the quality of individual components but also their proper integration into the final product. To assist in this matter, automated assembly control using computer vision methods has been widely implemented. However, the costs associated with image acquisition, annotation, and training of computer vision algorithms pose challenges for integration, especially for small- and medium-sized enterprises (SMEs), which often lack the resources for extensive training, data collection, and manual image annotation. Synthetic data offers the potential to reduce manual data collection and labeling. Nevertheless, its practical application in the context of assembly quality remains limited. In this work, we present a novel approach for easily integrable and data-efficient visual assembly control. Our approach leverages simulated scene generation based on computer-aided design (CAD) data and object detection algorithms. The results demonstrate a time-saving pipeline for generating image data in manufacturing environments, achieving a mean Average Precision (mAP@0.5:0.95) up to 99,5% for correctly identifying instances of synthetic planetary gear system components within our simulated training data, and up to 93% when transferred to real-world camera-captured testing data. This research highlights the effectiveness of synthetic data generation within an adaptable pipeline and underscores its potential to support SMEs in implementing resource-efficient visual assembly control solutions.
Abstract (translated)
装配过程中的质量控制在制造业中至关重要，不仅确保了各个组件的质量，还保证了它们能够正确地集成到最终产品中。为了协助这一目标，广泛实施了基于计算机视觉方法的自动化装配控制系统。然而，图像获取、注释以及训练计算机视觉算法的相关成本对整合带来了挑战，尤其是对于资源有限的小型和中型企业（SMEs），这些企业通常缺乏进行大量培训、数据收集及手动图像标注的能力。合成数据提供了减少手动数据采集和标记潜力的可能性，但其在装配质量上下文中的实际应用仍然较为有限。 在此研究工作中，我们提出了一种新型的方法，旨在实现易于集成且高效的数据驱动视觉装配控制。我们的方法利用基于计算机辅助设计（CAD）数据的仿真场景生成，并结合了目标检测算法。实验结果表明，这为制造业环境中图像数据的生成提供了一个节省时间的工作流程，在针对合成行星齿轮系统组件实例的模拟训练数据上实现了高达99.5%的平均精度（mAP@0.5:0.95），而在转移到实际相机捕捉到的测试数据时，这一数字达到了最高93%。 这项研究强调了在具有适应性工作流程中生成合成数据的有效性，并突显了其支持SMEs实施资源高效视觉装配控制系统解决方案的巨大潜力。
URL
https://arxiv.org/abs/2509.13089
PDF
-
Narrative-to-Scene Generation: An LLM-Driven Pipeline for 2D Game Environments
2025-08-31 01:45:56Yi-Chun Chen, Arnav JhalaAbstract
Recent advances in large language models(LLMs) enable compelling story generation, but connecting narrative text to playable visual environments remains an open challenge in procedural content generation(PCG). We present a lightweight pipeline that transforms short narrative prompts into a sequence of 2D tile-based game scenes, reflecting the temporal structure of stories. Given an LLM-generated narrative, our system identifies three key time frames, extracts spatial predicates in the form of "Object-Relation-Object" triples, and retrieves visual assets using affordance-aware semantic embeddings from the GameTileNet dataset. A layered terrain is generated using Cellular Automata, and objects are placed using spatial rules grounded in the predicate structure. We evaluated our system in ten diverse stories, analyzing tile-object matching, affordance-layer alignment, and spatial constraint satisfaction across frames. This prototype offers a scalable approach to narrative-driven scene generation and lays the foundation for future work on multi-frame continuity, symbolic tracking, and multi-agent coordination in story-centered PCG.
Abstract (translated)
最近在大型语言模型（LLMs）方面的进展使得生成引人入胜的故事成为可能，但将叙述文本与可玩的视觉环境联系起来仍然是程序化内容生成（PCG）中的一个开放性挑战。我们提出了一种轻量级流水线，能够将简短的故事提示转换为一系列2D基于瓷砖的游戏场景，这些场景反映了故事的时间结构。 给定由大型语言模型生成的叙述文本后，我们的系统会识别出三个关键时间框架，并提取空间谓词的形式化为“对象-关系-对象”三元组。然后使用带有GameTileNet数据集中感知功能的语义嵌入来检索视觉资产。地形通过细胞自动机分层生成，而根据谓词结构的空间规则放置物体。 我们在十种不同类型的故事情节中评估了该系统，分析了瓷砖与物体之间的匹配度、功能层次的一致性以及跨框架的空间约束满足情况。 这一原型提供了一种基于叙事驱动场景生成的可扩展方法，并为未来工作奠定了基础，这些未来的工作将涉及多帧连续性、符号跟踪和以故事为中心的多智能体协调。
URL
https://arxiv.org/abs/2509.04481
PDF
-
LSD-3D: Large-Scale 3D Driving Scene Generation with Geometry Grounding
2025-08-26 17:04:49Julian Ost, Andrea Ramazzina, Amogh Joshi, Maximilian B\"omer, Mario Bijelic, Felix HeideAbstract
Large-scale scene data is essential for training and testing in robot learning. Neural reconstruction methods have promised the capability of reconstructing large physically-grounded outdoor scenes from captured sensor data. However, these methods have baked-in static environments and only allow for limited scene control -- they are functionally constrained in scene and trajectory diversity by the captures from which they are reconstructed. In contrast, generating driving data with recent image or video diffusion models offers control, however, at the cost of geometry grounding and causality. In this work, we aim to bridge this gap and present a method that directly generates large-scale 3D driving scenes with accurate geometry, allowing for causal novel view synthesis with object permanence and explicit 3D geometry estimation. The proposed method combines the generation of a proxy geometry and environment representation with score distillation from learned 2D image priors. We find that this approach allows for high controllability, enabling the prompt-guided geometry and high-fidelity texture and structure that can be conditioned on map layouts -- producing realistic and geometrically consistent 3D generations of complex driving scenes.
Abstract (translated)
大规模场景数据对于机器人学习的训练和测试至关重要。神经重建方法承诺可以从捕获的传感器数据中重构大型且物理基础坚实的室外场景。然而，这些方法内置了静态环境限制，并仅允许有限的场景控制——它们的功能受限于用于重建的数据捕捉中的多样性和轨迹多样性。 相比之下，使用最近的图像或视频扩散模型生成驾驶数据可以提供更多的控制能力，但代价是失去了几何精确度和因果关系。在这项工作中，我们旨在弥合这一差距，并提出了一种直接生成大规模3D驾驶场景的方法，该方法具有准确的几何形状，并且能够进行因果驱动的新视图合成、物体持久性和显式的3D几何估计。 所提出的这种方法结合了代理几何体和环境表示的生成以及从学习到的2D图像先验中提取分数。我们发现，这种方法允许高度可控性，可以基于地图布局对提示引导下的几何形状和高保真的纹理及结构进行条件设定——从而能够产生现实且几何上一致的大规模驾驶场景的3D生成。
URL
https://arxiv.org/abs/2508.19204
PDF
-
HLG: Comprehensive 3D Room Construction via Hierarchical Layout Generation
2025-08-25 09:32:57Xiping Wang, Yuxi Wang, Mengqi Zhou, Junsong Fan, Zhaoxiang ZhangAbstract
Realistic 3D indoor scene generation is crucial for virtual reality, interior design, embodied intelligence, and scene understanding. While existing methods have made progress in coarse-scale furniture arrangement, they struggle to capture fine-grained object placements, limiting the realism and utility of generated environments. This gap hinders immersive virtual experiences and detailed scene comprehension for embodied AI applications. To address these issues, we propose Hierarchical Layout Generation (HLG), a novel method for fine-grained 3D scene generation. HLG is the first to adopt a coarse-to-fine hierarchical approach, refining scene layouts from large-scale furniture placement to intricate object arrangements. Specifically, our fine-grained layout alignment module constructs a hierarchical layout through vertical and horizontal decoupling, effectively decomposing complex 3D indoor scenes into multiple levels of granularity. Additionally, our trainable layout optimization network addresses placement issues, such as incorrect positioning, orientation errors, and object intersections, ensuring structurally coherent and physically plausible scene generation. We demonstrate the effectiveness of our approach through extensive experiments, showing superior performance in generating realistic indoor scenes compared to existing methods. This work advances the field of scene generation and opens new possibilities for applications requiring detailed 3D environments. We will release our code upon publication to encourage future research.
Abstract (translated)
生成现实主义的三维室内场景对于虚拟现实、室内设计、具身智能和场景理解至关重要。尽管现有的方法在粗略规模的家具布局方面已取得进展，但在捕捉细微的对象放置细节上仍存在困难，这限制了生成环境的真实性和实用性。这种差距阻碍了沉浸式虚拟体验的发展以及对复杂AI应用场景中详细场景的理解。为了解决这些问题，我们提出了一种名为层级布局生成（HLG）的新方法，用于精细级的3D场景生成。HLG首次采用了一个从粗到细的分层方式，通过细化大型家具放置来实现复杂的对象安排布局。 具体而言，我们的精细化布局对齐模块通过对垂直和水平方向进行解耦来构建一个层级式布局，有效地将复杂的三维室内场景分解为多个级别的详细程度。此外，我们可训练的布局优化网络解决了摆放问题，例如不正确的定位、方向错误以及物体之间的碰撞，确保生成的场景在结构上是连贯且物理上可信的。 通过广泛的实验，我们展示了这种方法的有效性，在生成逼真的室内场景方面优于现有的方法。这项工作推动了场景生成领域的发展，并为需要详细3D环境的应用开辟了新的可能性。我们将会在发表时发布我们的代码以促进未来的相关研究。
URL
https://arxiv.org/abs/2508.17832
PDF
-
FAIRGAMER: Evaluating Biases in the Application of Large Language Models to Video Games
2025-08-25 09:26:19Bingkang Shi, Jen-tse Huang, Guoyi Li, Xiaodan Zhang, Zhongjiang YaoAbstract
Leveraging their advanced capabilities, Large Language Models (LLMs) demonstrate vast application potential in video games--from dynamic scene generation and intelligent NPC interactions to adaptive opponents--replacing or enhancing traditional game mechanics. However, LLMs' trustworthiness in this application has not been sufficiently explored. In this paper, we reveal that the models' inherent social biases can directly damage game balance in real-world gaming environments. To this end, we present FairGamer, the first bias evaluation Benchmark for LLMs in video game scenarios, featuring six tasks and a novel metrics ${D_lstd}$. It covers three key scenarios in games where LLMs' social biases are particularly likely to manifest: Serving as Non-Player Characters, Interacting as Competitive Opponents, and Generating Game Scenes. FairGamer utilizes both reality-grounded and fully fictional game content, covering a variety of video game genres. Experiments reveal: (1) Decision biases directly cause game balance degradation, with Grok-3 (average ${D_lstd}$ score=0.431) exhibiting the most severe degradation; (2) LLMs demonstrate isomorphic social/cultural biases toward both real and virtual world content, suggesting their biases nature may stem from inherent model characteristics. These findings expose critical reliability gaps in LLMs' gaming applications. Our code and data are available at anonymous GitHub this https URL .
Abstract (translated)
利用其先进的能力，大型语言模型（LLMs）在视频游戏中的应用潜力巨大——从动态场景生成、智能NPC互动到适应性对手，这些都可能替代或增强传统的游戏机制。然而，人们对LLM在这种应用场景中的可信度尚未进行充分探讨。在这篇论文中，我们揭示了模型内在的社会偏见可以直接损害实际游戏环境中的平衡性。为此，我们介绍了FairGamer——首个用于评估视频游戏中LLMs社会偏见的基准测试工具，该工具包括六个任务和一种新的度量指标${D_lstd}$。它涵盖了三个在游戏场景中LLM的社会偏见特别容易表现出来的关键领域：作为非玩家角色、充当竞争对手以及生成游戏场景。FairGamer同时使用了基于现实和完全虚构的游戏内容，涵盖多种视频游戏类型。 实验结果表明： 1. 决策偏差直接导致游戏平衡的恶化，其中Grok-3（平均${D_lstd}$分数=0.431）表现最差。 2. LLM对于真实世界和虚拟世界的主题均展示出同构的社会/文化偏见，这表明它们的偏见可能源自模型本身的特性。这些发现揭示了LLMs在游戏应用中的关键可靠性差距。 我们的代码和数据可在匿名GitHub（此URL不可直接链接，请访问论文获取具体网址）上获得。
URL
https://arxiv.org/abs/2508.17825
PDF
-
SceneGen: Single-Image 3D Scene Generation in One Feedforward Pass
2025-08-21 17:59:16Yanxu Meng, Haoning Wu, Ya Zhang, Weidi XieAbstract
3D content generation has recently attracted significant research interest due to its applications in VR/AR and embodied AI. In this work, we address the challenging task of synthesizing multiple 3D assets within a single scene image. Concretely, our contributions are fourfold: (i) we present SceneGen, a novel framework that takes a scene image and corresponding object masks as input, simultaneously producing multiple 3D assets with geometry and texture. Notably, SceneGen operates with no need for optimization or asset retrieval; (ii) we introduce a novel feature aggregation module that integrates local and global scene information from visual and geometric encoders within the feature extraction module. Coupled with a position head, this enables the generation of 3D assets and their relative spatial positions in a single feedforward pass; (iii) we demonstrate SceneGen's direct extensibility to multi-image input scenarios. Despite being trained solely on single-image inputs, our architectural design enables improved generation performance with multi-image inputs; and (iv) extensive quantitative and qualitative evaluations confirm the efficiency and robust generation abilities of our approach. We believe this paradigm offers a novel solution for high-quality 3D content generation, potentially advancing its practical applications in downstream tasks. The code and model will be publicly available at: this https URL.
Abstract (translated)
最近，由于在虚拟现实（VR）、增强现实（AR）和具身人工智能中的应用，三维内容生成引起了广泛的研究兴趣。在这项工作中，我们解决了一个具有挑战性的任务：从单个场景图像中合成多个三维资产。具体来说，我们的贡献有四个方面： (i) 我们提出了SceneGen，一个新颖的框架，它接受场景图像及其对应对象掩码作为输入，并同时产生带有几何形状和纹理的多个三维资产。值得注意的是，SceneGen无需优化或资产检索即可运行； (ii) 我们引入了一个新颖的功能聚合模块，在特征提取模块中整合了来自视觉和几何编码器的局部和全局场景信息。与位置头相结合，这使得在一个前馈传递过程中生成三维资产及其相对空间位置成为可能； (iii) 我们展示了SceneGen在多图像输入场景中的直接扩展性。尽管仅使用单个图像输入进行训练，但我们的架构设计允许使用多图像输入来提高生成性能； (iv) 全面的定量和定性评估确认了我们方法的有效性和鲁棒生成能力。 我们认为这一范式为高质量三维内容生成提供了新颖解决方案，并有可能推进其在下游任务中的实际应用。代码和模型将在以下网址公开发布：[此URL]。
URL
https://arxiv.org/abs/2508.15769
PDF
-
MeSS: City Mesh-Guided Outdoor Scene Generation with Cross-View Consistent Diffusion
2025-08-21 02:16:15Xuyang Chen, Zhijun Zhai, Kaixuan Zhou, Zengmao Wang, Jianan He, Dong Wang, Yanfeng Zhang, mingwei Sun, R\"udiger Westermann, Konrad Schindler, Liqiu MengAbstract
Mesh models have become increasingly accessible for numerous cities; however, the lack of realistic textures restricts their application in virtual urban navigation and autonomous driving. To address this, this paper proposes MeSS (Meshbased Scene Synthesis) for generating high-quality, styleconsistent outdoor scenes with city mesh models serving as the geometric prior. While image and video diffusion models can leverage spatial layouts (such as depth maps or HD maps) as control conditions to generate street-level perspective views, they are not directly applicable to 3D scene generation. Video diffusion models excel at synthesizing consistent view sequences that depict scenes but often struggle to adhere to predefined camera paths or align accurately with rendered control videos. In contrast, image diffusion models, though unable to guarantee cross-view visual consistency, can produce more geometry-aligned results when combined with ControlNet. Building on this insight, our approach enhances image diffusion models by improving cross-view consistency. The pipeline comprises three key stages: first, we generate geometrically consistent sparse views using Cascaded Outpainting ControlNets; second, we propagate denser intermediate views via a component dubbed AGInpaint; and third, we globally eliminate visual inconsistencies (e.g., varying exposure) using the GCAlign module. Concurrently with generation, a 3D Gaussian Splatting (3DGS) scene is reconstructed by initializing Gaussian balls on the mesh surface. Our method outperforms existing approaches in both geometric alignment and generation quality. Once synthesized, the scene can be rendered in diverse styles through relighting and style transfer techniques.
Abstract (translated)
网格模型（Mesh models）对于许多城市来说已经变得越来越容易获取；然而，缺乏逼真的纹理限制了它们在虚拟城市导航和自动驾驶中的应用。为了应对这一问题，本文提出了 MeSS (基于网格的场景合成)，旨在利用城市的网格模型作为几何先验条件来生成高质量、风格一致的户外场景。虽然图像和视频扩散模型可以借助空间布局（如深度图或高清地图）作为控制条件来生成街道视角视图，但它们并不直接适用于3D场景生成。视频扩散模型擅长于合成描绘场景的一致性视图序列，但在遵循预定义的相机路径或与渲染的控制视频对齐方面常常遇到困难。相比之下，尽管图像扩散模型无法保证跨视角视觉一致性，但如果结合使用ControlNet，则可以产生更具几何一致性的结果。基于这一洞察，我们的方法通过改进图像扩散模型之间的视角一致性来增强它们的功能。 该流程包含三个关键阶段：首先，我们利用级联Outpainting ControlNets生成几何上一致的稀疏视图；其次，我们将更密集的中间视图通过名为AGInpaint的组件进行传播；第三，我们使用GCAlign模块全局消除视觉不一致性（例如不同的曝光度）。与此同时，在生成过程中，通过在网格表面上初始化高斯球来重建3D高斯点阵（3DGS）场景。我们的方法在几何对齐和生成质量方面都优于现有的方法。一旦合成完成，可以通过重新照明和风格转移技术以多种样式渲染场景。 这种综合方法不仅提升了基于图像扩散模型的三维场景生成能力，还为未来在虚拟城市导航和自动驾驶领域中的应用铺平了道路，使得利用现有网格数据创建逼真的虚拟环境成为可能。
URL
https://arxiv.org/abs/2508.15169
PDF
-
TiP4GEN: Text to Immersive Panorama 4D Scene Generation
2025-08-17 16:02:24Ke Xing, Hanwen Liang, Dejia Xu, Yuyang Yin, Konstantinos N. Plataniotis, Yao Zhao, Yunchao WeiAbstract
With the rapid advancement and widespread adoption of VR/AR technologies, there is a growing demand for the creation of high-quality, immersive dynamic scenes. However, existing generation works predominantly concentrate on the creation of static scenes or narrow perspective-view dynamic scenes, falling short of delivering a truly 360-degree immersive experience from any viewpoint. In this paper, we introduce \textbf{TiP4GEN}, an advanced text-to-dynamic panorama scene generation framework that enables fine-grained content control and synthesizes motion-rich, geometry-consistent panoramic 4D scenes. TiP4GEN integrates panorama video generation and dynamic scene reconstruction to create 360-degree immersive virtual environments. For video generation, we introduce a \textbf{Dual-branch Generation Model} consisting of a panorama branch and a perspective branch, responsible for global and local view generation, respectively. A bidirectional cross-attention mechanism facilitates comprehensive information exchange between the branches. For scene reconstruction, we propose a \textbf{Geometry-aligned Reconstruction Model} based on 3D Gaussian Splatting. By aligning spatial-temporal point clouds using metric depth maps and initializing scene cameras with estimated poses, our method ensures geometric consistency and temporal coherence for the reconstructed scenes. Extensive experiments demonstrate the effectiveness of our proposed designs and the superiority of TiP4GEN in generating visually compelling and motion-coherent dynamic panoramic scenes. Our project page is at this https URL.
Abstract (translated)
随着虚拟现实（VR）和增强现实（AR）技术的迅速发展及其广泛应用，对高质量、沉浸式动态场景的需求日益增长。然而，现有的生成工作主要集中在静态场景或窄视角动态场景的创建上，无法提供从任意视点出发的真实360度沉浸式体验。在本文中，我们介绍了**TiP4GEN**，这是一个先进的文本到动态全景场景生成框架，能够实现精细的内容控制，并合成富于动作、几何一致性的全景4D场景。TiP4GEN结合了全景视频生成和动态场景重建技术，以创建360度的沉浸式虚拟环境。 对于视频生成部分，我们提出了一种**双分支生成模型**，包括一个用于全局视图生成的全景支路和一个用于局部视图生成的透视支路。双向交叉注意机制促进了两个分支之间全面的信息交换。 在场景重建方面，我们提出了基于三维高斯点云的**几何对齐重构模型**。通过使用度量深度地图来对齐时空点云，并利用估计的姿态初始化场景摄像机，我们的方法确保了重建场景的几何一致性和时间连贯性。 广泛的实验验证了我们所提出的方案的有效性和TiP4GEN在生成视觉吸引力强、动作协调的动态全景场景方面的优越性。我们的项目页面位于此[URL]（请将此处的"this https URL"替换为实际链接）。
URL
https://arxiv.org/abs/2508.12415
PDF
-
ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Driving
2025-08-15 12:06:55Jingyu Li, Bozhou Zhang, Xin Jin, Jiankang Deng, Xiatian Zhu, Li ZhangAbstract
Autonomous driving requires rich contextual comprehension and precise predictive reasoning to navigate dynamic and complex environments safely. Vision-Language Models (VLMs) and Driving World Models (DWMs) have independently emerged as powerful recipes addressing different aspects of this challenge. VLMs provide interpretability and robust action prediction through their ability to understand multi-modal context, while DWMs excel in generating detailed and plausible future driving scenarios essential for proactive planning. Integrating VLMs with DWMs is an intuitive, promising, yet understudied strategy to exploit the complementary strengths of accurate behavioral prediction and realistic scene generation. Nevertheless, this integration presents notable challenges, particularly in effectively connecting action-level decisions with high-fidelity pixel-level predictions and maintaining computational efficiency. In this paper, we propose ImagiDrive, a novel end-to-end autonomous driving framework that integrates a VLM-based driving agent with a DWM-based scene imaginer to form a unified imagination-and-planning loop. The driving agent predicts initial driving trajectories based on multi-modal inputs, guiding the scene imaginer to generate corresponding future scenarios. These imagined scenarios are subsequently utilized to iteratively refine the driving agent's planning decisions. To address efficiency and predictive accuracy challenges inherent in this integration, we introduce an early stopping mechanism and a trajectory selection strategy. Extensive experimental validation on the nuScenes and NAVSIM datasets demonstrates the robustness and superiority of ImagiDrive over previous alternatives under both open-loop and closed-loop conditions.
Abstract (translated)
自动驾驶需要丰富的语境理解和精确的预测推理，以安全地导航动态和复杂的环境。视觉语言模型（VLMs）和驾驶世界模型（DWMs）分别在解决这一挑战的不同方面中独立涌现为强有力的解决方案。VLMs通过其理解多模态上下文的能力提供解释性和稳健的动作预测，而DWMs则擅长生成详细且可信的未来驾驶场景，这对于主动规划至关重要。将VLM与DWM结合是一种直观、有前景但研究不足的战略，旨在利用准确的行为预测和逼真场景生成的互补优势。然而，这种整合带来了显著挑战，特别是在有效地连接行动级决策与高保真的像素级预测以及保持计算效率方面。 在本文中，我们提出了ImagiDrive——一种新颖的端到端自动驾驶框架，它将基于VLM的驾驶代理与基于DWM的场景想象器相结合，形成统一的想象和规划循环。该驾驶代理根据多模态输入预测初始行驶轨迹，并指导场景想象器生成相应的未来情景。这些被设想的情景随后用于迭代优化驾驶代理的规划决策。 为了应对这种整合中固有的效率和预测准确性挑战，我们引入了一个早期停止机制和一种轨迹选择策略。在nuScenes和NAVSIM数据集上的广泛实验验证表明，在开环和闭环条件下，ImagiDrive比以前的方法具有更强健性和优越性。
URL
https://arxiv.org/abs/2508.11428
PDF
-
HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editing
2025-08-07 23:23:07Zixuan Bian, Ruohan Ren, Yue Yang, Chris Callison-BurchAbstract
3D scene generation plays a crucial role in gaming, artistic creation, virtual reality and many other domains. However, current 3D scene design still relies heavily on extensive manual effort from creators, and existing automated methods struggle to generate open-domain scenes or support flexible editing. As a result, generating 3D worlds directly from text has garnered increasing attention. In this paper, we introduce HOLODECK 2.0, an advanced vision-language-guided framework for 3D world generation with support for interactive scene editing based on human feedback. HOLODECK 2.0 can generate diverse and stylistically rich 3D scenes (e.g., realistic, cartoon, anime, and cyberpunk styles) that exhibit high semantic fidelity to fine-grained input descriptions, suitable for both indoor and open-domain environments. HOLODECK 2.0 leverages vision-language models (VLMs) to identify and parse the objects required in a scene and generates corresponding high-quality assets via state-of-the-art 3D generative models. It then iteratively applies spatial constraints derived from the VLMs to achieve semantically coherent and physically plausible layouts. Human evaluations and CLIP-based assessments demonstrate that HOLODECK 2.0 effectively generates high-quality scenes closely aligned with detailed textual descriptions, consistently outperforming baselines across indoor and open-domain scenarios. Additionally, we provide editing capabilities that flexibly adapt to human feedback, supporting layout refinement and style-consistent object edits. Finally, we present a practical application of HOLODECK 2.0 in procedural game modeling, generating visually rich and immersive environments, potentially boosting efficiency.
Abstract (translated)
3D场景生成在游戏、艺术创作、虚拟现实等多个领域中扮演着至关重要的角色。然而，当前的3D场景设计仍然高度依赖于创作者大量的手工劳动，并且现有的自动化方法难以生成开放领域的场景或支持灵活编辑。因此，从文本直接生成3D世界引起了越来越多的关注。在这篇论文中，我们介绍了HOLODECK 2.0，这是一个先进的基于视觉-语言指导的框架，用于生成具有互动场景编辑功能（基于人类反馈）的3D世界。HOLODECK 2.0能够生成多样化且风格丰富的3D场景（例如：写实、卡通、动漫和赛博朋克风格），这些场景与细粒度输入描述的高度语义一致，适用于室内及开放领域环境。通过利用视觉-语言模型(VLMs)，HOLODECK 2.0可以识别并解析场景所需的对象，并借助最先进的3D生成模型来生成相应的高质量资产。然后，它会迭代应用由VLMs推导的空间约束，以实现语义连贯且物理上合理的布局。人类评估和CLIP（对比语言-图像预训练）评估表明，HOLODECK 2.0能够有效生成与详细文本描述高度一致的高质量场景，在室内及开放领域场景中持续优于基准方法。此外，我们提供了灵活适应人类反馈的编辑功能，支持布局调整和风格一致性对象编辑。最后，我们将展示HOLODECK 2.0在程序化游戏建模中的实际应用，生成视觉丰富且沉浸式的环境，有望提高效率。
URL
https://arxiv.org/abs/2508.05899
PDF
