Scene_Generation
-
ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary
2025-05-31 23:03:54Zeqi Gu, Yin Cui, Zhaoshuo Li, Fangyin Wei, Yunhao Ge, Jinwei Gu, Ming-Yu Liu, Abe Davis, Yifan DingAbstract
Designing 3D scenes is traditionally a challenging task that demands both artistic expertise and proficiency with complex software. Recent advances in text-to-3D generation have greatly simplified this process by letting users create scenes based on simple text descriptions. However, as these methods generally require extra training or in-context learning, their performance is often hindered by the limited availability of high-quality 3D data. In contrast, modern text-to-image models learned from web-scale images can generate scenes with diverse, reliable spatial layouts and consistent, visually appealing styles. Our key insight is that instead of learning directly from 3D scenes, we can leverage generated 2D images as an intermediary to guide 3D synthesis. In light of this, we introduce ArtiScene, a training-free automated pipeline for scene design that integrates the flexibility of free-form text-to-image generation with the diversity and reliability of 2D intermediary layouts. First, we generate 2D images from a scene description, then extract the shape and appearance of objects to create 3D models. These models are assembled into the final scene using geometry, position, and pose information derived from the same intermediary image. Being generalizable to a wide range of scenes and styles, ArtiScene outperforms state-of-the-art benchmarks by a large margin in layout and aesthetic quality by quantitative metrics. It also averages a 74.89% winning rate in extensive user studies and 95.07% in GPT-4o evaluation. Project page: this https URL
Abstract (translated)
设计三维场景传统上是一项既需要艺术专长又需掌握复杂软件技能的挑战性任务。最近在文本到3D生成领域的进步通过让用户基于简单的文字描述来创建场景，大大简化了这一过程。然而，由于这些方法通常要求额外训练或上下文学习，因此受限于高质量三维数据有限可用性的性能问题仍然存在。相比之下，现代从网络规模图像中学习的文本到图像模型能够产生具有多样性和可靠性空间布局以及一致且视觉吸引人的风格的场景。 我们的关键见解是：与其直接从3D场景进行学习，不如利用生成的2D图像作为中间体来指导3D合成。基于此理念，我们介绍了ArtiScene——一个无需训练的自动化管线，用于场景设计，该管道将自由形式文本到图像生成的灵活性与2D中间布局的多样性和可靠性相结合。 首先，从场景描述中生成2D图像；然后提取对象的形状和外观以创建3D模型。这些模型利用来自同一中间图像的几何、位置和姿态信息进行最终场景组装。ArtiScene能够广泛适用于各种类型的场景和风格，并且在广泛的用户研究中获得了74.89%的胜率，在GPT-4o评估中得到了95.07%的好评。 通过定量指标，ArtiScene在布局和美学质量上大大优于最先进的基准测试。项目页面：[提供链接]
URL
https://arxiv.org/abs/2506.00742
PDF
-
SPIRAL: Semantic-Aware Progressive LiDAR Scene Generation
2025-05-28 17:55:35Dekai Zhu, Yixuan Hu, Youquan Liu, Dongyue Lu, Lingdong Kong, Slobodan IlicAbstract
Leveraging recent diffusion models, LiDAR-based large-scale 3D scene generation has achieved great success. While recent voxel-based approaches can generate both geometric structures and semantic labels, existing range-view methods are limited to producing unlabeled LiDAR scenes. Relying on pretrained segmentation models to predict the semantic maps often results in suboptimal cross-modal consistency. To address this limitation while preserving the advantages of range-view representations, such as computational efficiency and simplified network design, we propose Spiral, a novel range-view LiDAR diffusion model that simultaneously generates depth, reflectance images, and semantic maps. Furthermore, we introduce novel semantic-aware metrics to evaluate the quality of the generated labeled range-view data. Experiments on the SemanticKITTI and nuScenes datasets demonstrate that Spiral achieves state-of-the-art performance with the smallest parameter size, outperforming two-step methods that combine the generative and segmentation models. Additionally, we validate that range images generated by Spiral can be effectively used for synthetic data augmentation in the downstream segmentation training, significantly reducing the labeling effort on LiDAR data.
Abstract (translated)
利用最近的扩散模型，基于LiDAR的大规模3D场景生成取得了显著的成功。尽管近期基于体素的方法可以同时生成几何结构和语义标签，现有的范围视图方法仅限于产生未标记的LiDAR场景。依赖预训练分割模型来预测语义地图通常会导致跨模态一致性较差的结果。为了在保留范围视图表示的优点（如计算效率和简化网络设计）的同时解决这一局限性，我们提出了Spiral，这是一种新颖的范围视图LiDAR扩散模型，可以同时生成深度图像、反射率图像和语义地图。此外，我们引入了新的基于语义的度量标准来评估生成的带标签的范围视图数据的质量。在SemanticKITTI和nuScenes数据集上的实验表明，Spiral实现了最先进的性能，并且参数规模最小，在结合生成模型和分割模型的两步方法中表现更佳。此外，我们验证了由Spiral生成的范围图像可以有效地用于下游分割训练中的合成数据增强，从而显著减少了对LiDAR数据的手动标注工作。
URL
https://arxiv.org/abs/2505.22643
PDF
-
Agentic 3D Scene Generation with Spatially Contextualized VLMs
2025-05-26 15:28:17Xinhang Liu, Yu-Wing Tai, Chi-Keung TangAbstract
Despite recent advances in multimodal content generation enabled by vision-language models (VLMs), their ability to reason about and generate structured 3D scenes remains largely underexplored. This limitation constrains their utility in spatially grounded tasks such as embodied AI, immersive simulations, and interactive 3D applications. We introduce a new paradigm that enables VLMs to generate, understand, and edit complex 3D environments by injecting a continually evolving spatial context. Constructed from multimodal input, this context consists of three components: a scene portrait that provides a high-level semantic blueprint, a semantically labeled point cloud capturing object-level geometry, and a scene hypergraph that encodes rich spatial relationships, including unary, binary, and higher-order constraints. Together, these components provide the VLM with a structured, geometry-aware working memory that integrates its inherent multimodal reasoning capabilities with structured 3D understanding for effective spatial reasoning. Building on this foundation, we develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context. The pipeline features high-quality asset generation with geometric restoration, environment setup with automatic verification, and ergonomic adjustment guided by the scene hypergraph. Experiments show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work. Further results demonstrate that injecting spatial context enables VLMs to perform downstream tasks such as interactive scene editing and path planning, suggesting strong potential for spatially intelligent systems in computer graphics, 3D vision, and embodied applications.
Abstract (translated)
尽管近期得益于视觉-语言模型（VLM）的进展，多模态内容生成得到了显著提升，但这些模型在理解和生成结构化的三维场景方面的能力仍然很大程度上未被探索。这一限制制约了它们在基于空间的任务中的实用性，如具身人工智能、沉浸式模拟和交互式的3D应用中。为此，我们引入了一个新的范例，使VLM能够通过注入不断演进的空间上下文来生成、理解和编辑复杂的三维环境。该上下文由多模态输入构建而成，并包含三个组成部分：场景肖像，提供高层次的语义蓝图；带有语义标签的点云，捕获对象级别的几何信息；以及场景超图，编码丰富的空间关系，包括一元、二元和高阶约束。这三部分共同为VLM提供了结构化且具有几何感知的工作记忆，将它的固有多模态推理能力与三维理解相结合，实现有效的空间推理。 在此基础上，我们开发了一个代理性3D场景生成流水线，在这个管道中，VLM可以迭代地读取并更新空间上下文。该流程的特点包括高质量的资产生成、带有自动验证的环境设置以及通过场景超图指导的人体工学调整。实验表明，我们的框架能够处理多样且具有挑战性的输入，达到了此前工作中未观察到的一般化水平。进一步的结果显示，注入空间上下文使VLM能够执行下游任务如交互式场景编辑和路径规划，这预示着在计算机图形、3D视觉以及具身应用中的智能空间系统有着强大的潜力。
URL
https://arxiv.org/abs/2505.20129
PDF
-
WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions
2025-05-23 17:59:24Zizhang Li, Hong-Xing Yu, Wei Liu, Yin Yang, Charles Herrmann, Gordon Wetzstein, Jiajun WuAbstract
WonderPlay is a novel framework integrating physics simulation with video generation for generating action-conditioned dynamic 3D scenes from a single image. While prior works are restricted to rigid body or simple elastic dynamics, WonderPlay features a hybrid generative simulator to synthesize a wide range of 3D dynamics. The hybrid generative simulator first uses a physics solver to simulate coarse 3D dynamics, which subsequently conditions a video generator to produce a video with finer, more realistic motion. The generated video is then used to update the simulated dynamic 3D scene, closing the loop between the physics solver and the video generator. This approach enables intuitive user control to be combined with the accurate dynamics of physics-based simulators and the expressivity of diffusion-based video generators. Experimental results demonstrate that WonderPlay enables users to interact with various scenes of diverse content, including cloth, sand, snow, liquid, smoke, elastic, and rigid bodies -- all using a single image input. Code will be made public. Project website: this https URL
Abstract (translated)
WonderPlay 是一个新颖的框架，它将物理模拟与视频生成相结合，可以从单一图像中生成条件化的动态3D场景。虽然先前的工作仅限于刚体或简单的弹性动力学，但 WonderPlay 特别设计了一个混合生成式仿真器来合成各种各样的 3D 动力学。该混合生成式仿真器首先使用物理求解器模拟粗糙的 3D 动力学，然后利用视频生成器在这些基础之上产生更精细、更具现实感的动作视频。接着，所生成的视频被用来更新动态的 3D 场景，形成一个闭环过程，在其中物理求解器与视频生成器之间相互作用。这种方案使得用户能够直观地控制场景，并且结合了基于物理模拟器的精确动力学和扩散基础视频生成器的表现力。 实验结果表明，WonderPlay 允许用户通过单一图像输入与包含布料、沙子、雪、液体、烟雾、弹性体以及刚性物体等不同内容的各种场景进行互动。代码将会公开发布。项目网站：[此链接](https://this.url/)
URL
https://arxiv.org/abs/2505.18151
PDF
-
CGS-GAN: 3D Consistent Gaussian Splatting GANs for High Resolution Human Head Synthesis
2025-05-23 07:56:25Florian Barthel, Wieland Morgenstern, Paul Hinzer, Anna Hilsmann, Peter EisertAbstract
Recently, 3D GANs based on 3D Gaussian splatting have been proposed for high quality synthesis of human heads. However, existing methods stabilize training and enhance rendering quality from steep viewpoints by conditioning the random latent vector on the current camera position. This compromises 3D consistency, as we observe significant identity changes when re-synthesizing the 3D head with each camera shift. Conversely, fixing the camera to a single viewpoint yields high-quality renderings for that perspective but results in poor performance for novel views. Removing view-conditioning typically destabilizes GAN training, often causing the training to collapse. In response to these challenges, we introduce CGS-GAN, a novel 3D Gaussian Splatting GAN framework that enables stable training and high-quality 3D-consistent synthesis of human heads without relying on view-conditioning. To ensure training stability, we introduce a multi-view regularization technique that enhances generator convergence with minimal computational overhead. Additionally, we adapt the conditional loss used in existing 3D Gaussian splatting GANs and propose a generator architecture designed to not only stabilize training but also facilitate efficient rendering and straightforward scaling, enabling output resolutions up to $2048^2$. To evaluate the capabilities of CGS-GAN, we curate a new dataset derived from FFHQ. This dataset enables very high resolutions, focuses on larger portions of the human head, reduces view-dependent artifacts for improved 3D consistency, and excludes images where subjects are obscured by hands or other objects. As a result, our approach achieves very high rendering quality, supported by competitive FID scores, while ensuring consistent 3D scene generation. Check our our project page here: this https URL
Abstract (translated)
最近，基于三维高斯点阵（3D Gaussian splatting）的三维生成对抗网络（3D GANs）被提出用于高质量的人脸合成。然而，现有的方法通过将随机潜在向量条件化于当前摄像机位置来稳定训练并提升从极端视角进行渲染的质量，这种方法会损害三维一致性，因为在重新合成人脸时随着相机的位置变化会产生明显的身份变化。相比之下，固定相机在一个单一的视角虽然可以为该视角提供高质量的渲染效果，但对于新的视图则表现不佳。移除视图条件化通常会导致GAN训练不稳定，经常使训练崩溃。 为了应对这些挑战，我们引入了CGS-GAN（Conditional Gaussian Splatting GAN），这是一种新型的三维高斯点阵生成对抗网络框架，它能够在不依赖于视图条件下实现稳定的训练和高质量、三维一致的人脸合成。为确保训练稳定性，我们提出了一种多视角正则化技术，该技术在几乎无额外计算成本的情况下增强了生成器的收敛性。此外，我们将现有的3D高斯点阵GAN中使用的条件损失进行了调整，并设计了一个新的生成器架构，不仅可以稳定训练过程，而且有助于高效的渲染和轻松扩展，从而能够输出高达$2048^2$分辨率的结果。 为了评估CGS-GAN的能力，我们基于FFHQ数据集构建了新的数据集。该数据集支持非常高的解析度，重点关注人脸的较大区域，并减少视图相关的伪影以改善三维一致性。同时排除了被手或其他物体遮挡的图像样本。最终结果表明，我们的方法在高质量渲染方面取得了卓越的成绩，这得到了竞争性的FID（Fréchet Inception Distance）分数的支持，同时也保证了一致的三维场景生成。 如需了解更多信息，请访问我们的项目页面：[此链接](https://this-url.com) （请将"this https URL"替换为您实际的网址）。
URL
https://arxiv.org/abs/2505.17590
PDF
-
Constructing a 3D Town from a Single Image
2025-05-21 17:10:47Kaizhi Zheng, Ruijian Zhang, Jing Gu, Jie Yang, Xin Eric WangAbstract
Acquiring detailed 3D scenes typically demands costly equipment, multi-view data, or labor-intensive modeling. Therefore, a lightweight alternative, generating complex 3D scenes from a single top-down image, plays an essential role in real-world applications. While recent 3D generative models have achieved remarkable results at the object level, their extension to full-scene generation often leads to inconsistent geometry, layout hallucinations, and low-quality meshes. In this work, we introduce 3DTown, a training-free framework designed to synthesize realistic and coherent 3D scenes from a single top-down view. Our method is grounded in two principles: region-based generation to improve image-to-3D alignment and resolution, and spatial-aware 3D inpainting to ensure global scene coherence and high-quality geometry generation. Specifically, we decompose the input image into overlapping regions and generate each using a pretrained 3D object generator, followed by a masked rectified flow inpainting process that fills in missing geometry while maintaining structural continuity. This modular design allows us to overcome resolution bottlenecks and preserve spatial structure without requiring 3D supervision or fine-tuning. Extensive experiments across diverse scenes show that 3DTown outperforms state-of-the-art baselines, including Trellis, Hunyuan3D-2, and TripoSG, in terms of geometry quality, spatial coherence, and texture fidelity. Our results demonstrate that high-quality 3D town generation is achievable from a single image using a principled, training-free approach.
Abstract (translated)
获取详细的三维场景通常需要昂贵的设备、多视角数据或复杂的建模过程。因此，一种轻量级的方法——从单一顶视图图像生成复杂三维场景，在实际应用中扮演着重要角色。尽管最近的三维生成模型在物体级别的表现非常出色，但它们扩展到全场景生成时往往会导致不一致的几何结构、布局幻觉以及低质量的网格。为此，我们在本研究中引入了3DTown，这是一个无需训练框架，旨在从单一顶视图图像合成真实且连贯的三维场景。 我们的方法基于两个原则：区域化生成以提高二维到三维的一致性和分辨率，以及空间感知的三维修复填充来确保全局场景的一致性及高质量几何结构生成。具体而言，我们将输入图像分解为重叠的区域，并使用预训练的三维对象生成器生成每个区域；随后进行掩码修正流修复填充过程，以填补缺失的几何信息并保持结构连续性。 这种模块化设计允许我们克服分辨率瓶颈问题，并在无需三维监督或微调的情况下保存空间结构。通过在各种场景中进行广泛的实验表明，3DTown 在几何质量、空间一致性以及纹理保真度方面均优于当前最佳基线模型（包括 Trellis, Hunyuan3D-2 和 TripoSG）。 我们的研究成果展示了基于单一图像生成高质量三维城镇的可行性，并且采用了一种无需训练的原理性方法。
URL
https://arxiv.org/abs/2505.15765
PDF
-
3D Scene Generation: A Survey
2025-05-08 17:59:54Beichen Wen, Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, Ziwei LiuAbstract
3D scene generation seeks to synthesize spatially structured, semantically meaningful, and photorealistic environments for applications such as immersive media, robotics, autonomous driving, and embodied AI. Early methods based on procedural rules offered scalability but limited diversity. Recent advances in deep generative models (e.g., GANs, diffusion models) and 3D representations (e.g., NeRF, 3D Gaussians) have enabled the learning of real-world scene distributions, improving fidelity, diversity, and view consistency. Recent advances like diffusion models bridge 3D scene synthesis and photorealism by reframing generation as image or video synthesis problems. This survey provides a systematic overview of state-of-the-art approaches, organizing them into four paradigms: procedural generation, neural 3D-based generation, image-based generation, and video-based generation. We analyze their technical foundations, trade-offs, and representative results, and review commonly used datasets, evaluation protocols, and downstream applications. We conclude by discussing key challenges in generation capacity, 3D representation, data and annotations, and evaluation, and outline promising directions including higher fidelity, physics-aware and interactive generation, and unified perception-generation models. This review organizes recent advances in 3D scene generation and highlights promising directions at the intersection of generative AI, 3D vision, and embodied intelligence. To track ongoing developments, we maintain an up-to-date project page: this https URL.
Abstract (translated)
三维场景生成旨在为沉浸式媒体、机器人技术、自动驾驶和具身人工智能等应用合成具有空间结构化、语义意义且逼真的环境。早期基于程序规则的方法虽然具备可扩展性，但多样性有限。近年来，深度生成模型（如GANs、扩散模型）以及3D表示方法（如NeRF、3D高斯分布）的进步使得能够学习真实场景的分布，从而提高了逼真度、多样性和视角一致性。最近的技术进步，比如扩散模型通过将生成问题重新定义为图像或视频合成问题的方式，成功地连接了三维场景生成与照片级真实性。本综述系统性地概述了当前最先进的方法，并将其分类为四大范式：程序化生成、基于神经3D的生成、基于图像的生成和基于视频的生成。我们分析了它们的技术基础、权衡以及代表性结果，还回顾了一些常用的数据库、评估协议及下游应用。最后，讨论了在生成能力、三维表示、数据与注释以及评估方面的关键挑战，并概述了一系列有前景的方向，包括更高的保真度、物理感知和互动生成，以及统一的感知-生成模型。这篇综述整理了最近在三维场景生成领域的进展，并突出了生成AI、三维视觉和具身智能交叉领域中的潜在发展方向。为了跟踪正在进行的发展，我们维护了一个最新的项目页面：[此URL](this https URL)。
URL
https://arxiv.org/abs/2505.05474
PDF
-
Steerable Scene Generation with Post Training and Inference-Time Search
2025-05-07 22:07:42Nicholas Pfaff, Hongkai Dai, Sergey Zakharov, Shun Iwase, Russ TedrakeAbstract
Training robots in simulation requires diverse 3D scenes that reflect the specific challenges of downstream tasks. However, scenes that satisfy strict task requirements, such as high-clutter environments with plausible spatial arrangement, are rare and costly to curate manually. Instead, we generate large-scale scene data using procedural models that approximate realistic environments for robotic manipulation, and adapt it to task-specific goals. We do this by training a unified diffusion-based generative model that predicts which objects to place from a fixed asset library, along with their SE(3) poses. This model serves as a flexible scene prior that can be adapted using reinforcement learning-based post training, conditional generation, or inference-time search, steering generation toward downstream objectives even when they differ from the original data distribution. Our method enables goal-directed scene synthesis that respects physical feasibility and scales across scene types. We introduce a novel MCTS-based inference-time search strategy for diffusion models, enforce feasibility via projection and simulation, and release a dataset of over 44 million SE(3) scenes spanning five diverse environments. Website with videos, code, data, and model weights: this https URL
Abstract (translated)
在仿真中训练机器人需要反映下游任务特定挑战的多样化三维场景。然而，满足严格任务要求（如具有合理空间布局的高杂乱环境）的场景非常罕见且难以手动整理。因此，我们利用过程模型生成大规模场景数据，这些模型可以近似现实环境中机器人的操作需求，并将这些数据调整以适应具体任务目标。为此，我们训练了一个统一的基于扩散的生成模型，该模型能预测从固定资产库中放置哪些物体及其 SE(3) 位姿（位置和姿态）。此模型作为一个灵活的场景先验知识可以利用强化学习后训练、条件生成或推理时间搜索进行调整，即使这些目标与原始数据分布不同也能引导生成朝向下游目标发展。我们的方法实现了尊重物理可行性的目标导向场景合成，并且能够跨各种场景类型扩展。 我们还引入了一种基于MCTS（蒙特卡罗树搜索）的推理时间搜索策略应用于扩散模型中，并通过投影和模拟来确保可行性，同时发布了一个包含超过4400万SE(3)场景的数据集，这些场景涵盖了五种多样的环境。网站提供了视频、代码、数据和模型权重：[此链接](https://example.com/)（请根据实际链接替换）。
URL
https://arxiv.org/abs/2505.04831
PDF
-
Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation
2025-05-05 17:59:58Lu Ling, Chen-Hsuan Lin, Tsung-Yi Lin, Yifan Ding, Yu Zeng, Yichen Sheng, Yunhao Ge, Ming-Yu Liu, Aniket Bera, Zhaoshuo LiAbstract
Synthesizing interactive 3D scenes from text is essential for gaming, virtual reality, and embodied AI. However, existing methods face several challenges. Learning-based approaches depend on small-scale indoor datasets, limiting the scene diversity and layout complexity. While large language models (LLMs) can leverage diverse text-domain knowledge, they struggle with spatial realism, often producing unnatural object placements that fail to respect common sense. Our key insight is that vision perception can bridge this gap by providing realistic spatial guidance that LLMs lack. To this end, we introduce Scenethesis, a training-free agentic framework that integrates LLM-based scene planning with vision-guided layout refinement. Given a text prompt, Scenethesis first employs an LLM to draft a coarse layout. A vision module then refines it by generating an image guidance and extracting scene structure to capture inter-object relations. Next, an optimization module iteratively enforces accurate pose alignment and physical plausibility, preventing artifacts like object penetration and instability. Finally, a judge module verifies spatial coherence. Comprehensive experiments show that Scenethesis generates diverse, realistic, and physically plausible 3D interactive scenes, making it valuable for virtual content creation, simulation environments, and embodied AI research.
Abstract (translated)
从文本合成交互式3D场景对于游戏、虚拟现实和具身人工智能（Embodied AI）至关重要。然而，现有的方法面临着一些挑战。基于学习的方法依赖于小型的室内数据集，限制了场景多样性和布局复杂性；而大型语言模型（LLM）虽然可以利用多样的文本领域知识，但在空间真实感方面表现不佳，常常生成不符合常识的物体放置位置。我们的关键洞察是视觉感知能够弥合这一差距，通过提供现实的空间指导来弥补大型语言模型在此方面的不足。 为此，我们引入了Scenethesis框架，这是一个无需训练的代理框架，将基于LLM的场景规划与由视觉引导的布局精炼相结合。给定一个文本提示后，Scenethesis首先使用LLM生成粗略的布局草案；然后通过产生图像指引并提取场景结构来捕捉物体间的关联性以进行细化处理。接下来，优化模块迭代执行精确的姿态对齐和物理合理性检查，防止诸如物体穿透或不稳定等伪影现象的发生。最后，评判模块验证空间一致性。 全面实验表明，Scenethesis能够生成多样、真实且符合物理规律的3D交互场景，在虚拟内容创建、模拟环境以及具身人工智能研究方面具有重要的价值。
URL
https://arxiv.org/abs/2505.02836
PDF
-
MCCD: Multi-Agent Collaboration-based Compositional Diffusion for Complex Text-to-Image Generation
2025-05-05 13:50:03Mingcheng Li, Xiaolu Hou, Ziyang Liu, Dingkang Yang, Ziyun Qian, Jiawei Chen, Jinjie Wei, Yue Jiang, Qingyao Xu, Lihua ZhangAbstract
Diffusion models have shown excellent performance in text-to-image generation. Nevertheless, existing methods often suffer from performance bottlenecks when handling complex prompts that involve multiple objects, characteristics, and relations. Therefore, we propose a Multi-agent Collaboration-based Compositional Diffusion (MCCD) for text-to-image generation for complex scenes. Specifically, we design a multi-agent collaboration-based scene parsing module that generates an agent system comprising multiple agents with distinct tasks, utilizing MLLMs to extract various scene elements effectively. In addition, Hierarchical Compositional diffusion utilizes a Gaussian mask and filtering to refine bounding box regions and enhance objects through region enhancement, resulting in the accurate and high-fidelity generation of complex scenes. Comprehensive experiments demonstrate that our MCCD significantly improves the performance of the baseline models in a training-free manner, providing a substantial advantage in complex scene generation.
Abstract (translated)
扩散模型在文本到图像生成方面表现出色。然而，现有的方法在处理涉及多个对象、特征和关系的复杂提示时常常会遇到性能瓶颈。因此，我们提出了一种基于多智能体协作的组合扩散（MCCD）方法，用于复杂的场景文本到图像生成。具体而言，我们设计了一个基于多智能体协作的场景解析模块，该模块生成一个由执行不同任务的多个代理组成的系统，并利用大规模语言模型有效地提取各种场景元素。此外，层次化组合扩散通过使用高斯掩码和过滤技术来细化边界框区域并增强对象，从而实现了复杂场景的精确且高质量生成。 全面的实验表明，在无需训练的情况下，我们的MCCD显著提高了基准模型在复杂场景生成方面的性能，并提供了明显的优势。
URL
https://arxiv.org/abs/2505.02648
PDF
-
DualDiff: Dual-branch Diffusion Model for Autonomous Driving with Semantic Fusion
2025-05-03 16:20:01Haoteng Li, Zhao Yang, Zezhong Qian, Gongpeng Zhao, Yuqi Huang, Jun Yu, Huazheng Zhou, Longjun LiuAbstract
Accurate and high-fidelity driving scene reconstruction relies on fully leveraging scene information as conditioning. However, existing approaches, which primarily use 3D bounding boxes and binary maps for foreground and background control, fall short in capturing the complexity of the scene and integrating multi-modal information. In this paper, we propose DualDiff, a dual-branch conditional diffusion model designed to enhance multi-view driving scene generation. We introduce Occupancy Ray Sampling (ORS), a semantic-rich 3D representation, alongside numerical driving scene representation, for comprehensive foreground and background control. To improve cross-modal information integration, we propose a Semantic Fusion Attention (SFA) mechanism that aligns and fuses features across modalities. Furthermore, we design a foreground-aware masked (FGM) loss to enhance the generation of tiny objects. DualDiff achieves state-of-the-art performance in FID score, as well as consistently better results in downstream BEV segmentation and 3D object detection tasks.
Abstract (translated)
准确且高保真的驾驶场景重建依赖于充分利用场景信息作为条件。然而，现有的方法主要依靠三维边界框和二值图来控制前景和背景，在捕捉场景复杂性和整合多模态信息方面存在不足。在本文中，我们提出了DualDiff，这是一种双分支条件扩散模型，旨在提升多视角驾驶场景生成的效果。我们引入了Occupancy Ray Sampling（ORS），一种语义丰富的三维表示方法，并结合数值驾驶场景表示，实现全面的前景和背景控制。为了改进跨模态信息整合，我们提出了一种语义融合注意力(SFA)机制，用于对齐并融合不同模态特征。此外，我们设计了一种基于前景感知的掩码（FGM）损失函数，以增强微小物体生成的效果。DualDiff在FID评分上达到了最先进的性能，并且在下游BEV分割和3D目标检测任务中也表现出一致更好的结果。
URL
https://arxiv.org/abs/2505.01857
PDF
-
HoloTime: Taming Video Diffusion Models for Panoramic 4D Scene Generation
2025-04-30 13:55:28Haiyang Zhou, Wangbo Yu, Jiawen Guan, Xinhua Cheng, Yonghong Tian, Li YuanAbstract
The rapid advancement of diffusion models holds the promise of revolutionizing the application of VR and AR technologies, which typically require scene-level 4D assets for user experience. Nonetheless, existing diffusion models predominantly concentrate on modeling static 3D scenes or object-level dynamics, constraining their capacity to provide truly immersive experiences. To address this issue, we propose HoloTime, a framework that integrates video diffusion models to generate panoramic videos from a single prompt or reference image, along with a 360-degree 4D scene reconstruction method that seamlessly transforms the generated panoramic video into 4D assets, enabling a fully immersive 4D experience for users. Specifically, to tame video diffusion models for generating high-fidelity panoramic videos, we introduce the 360World dataset, the first comprehensive collection of panoramic videos suitable for downstream 4D scene reconstruction tasks. With this curated dataset, we propose Panoramic Animator, a two-stage image-to-video diffusion model that can convert panoramic images into high-quality panoramic videos. Following this, we present Panoramic Space-Time Reconstruction, which leverages a space-time depth estimation method to transform the generated panoramic videos into 4D point clouds, enabling the optimization of a holistic 4D Gaussian Splatting representation to reconstruct spatially and temporally consistent 4D scenes. To validate the efficacy of our method, we conducted a comparative analysis with existing approaches, revealing its superiority in both panoramic video generation and 4D scene reconstruction. This demonstrates our method's capability to create more engaging and realistic immersive environments, thereby enhancing user experiences in VR and AR applications.
Abstract (translated)
扩散模型的迅速发展预示着其在虚拟现实（VR）和增强现实（AR）技术应用中的革命性潜力，这些领域通常需要场景级别的4D资产来提升用户体验。然而，现有的扩散模型主要集中在静态3D场景或对象级别动态建模上，这限制了它们提供真正沉浸式体验的能力。为了解决这个问题，我们提出了一种名为HoloTime的框架，该框架集成了视频扩散模型，可以从单个提示或参考图像生成全景视频，并结合一种无缝将生成的全景视频转化为4D资产的方法，从而实现用户全方位的沉浸式4D体验。 具体而言，为了使视频扩散模型能够生成高质量的全景视频，我们引入了360World数据集，这是第一个针对下游4D场景重建任务而设计的全面全景视频集合。借助这一精心策划的数据集，我们提出了Panoramic Animator，这是一种两阶段的图像到视频扩散模型，可以将全景图像转换为高质量的全景视频。 在此之后，我们介绍了全景时空重构方法，这种方法利用时空深度估计技术将生成的全景视频转化为4D点云，并优化整体4D高斯散布表示法来重建在空间和时间上都一致的4D场景。为了验证我们的方法的有效性，我们进行了与现有方法的比较分析，结果表明它在全景视频生成和4D场景重构方面均表现出卓越性能。 这证明了我们的方法能够创建更吸引人且更具现实感的沉浸式环境，从而提升VR和AR应用中的用户体验。
URL
https://arxiv.org/abs/2504.21650
PDF
-
CasaGPT: Cuboid Arrangement and Scene Assembly for Interior Design
2025-04-28 04:35:04Weitao Feng, Hang Zhou, Jing Liao, Li Cheng, Wenbo ZhouAbstract
We present a novel approach for indoor scene synthesis, which learns to arrange decomposed cuboid primitives to represent 3D objects within a scene. Unlike conventional methods that use bounding boxes to determine the placement and scale of 3D objects, our approach leverages cuboids as a straightforward yet highly effective alternative for modeling objects. This allows for compact scene generation while minimizing object intersections. Our approach, coined CasaGPT for Cuboid Arrangement and Scene Assembly, employs an autoregressive model to sequentially arrange cuboids, producing physically plausible scenes. By applying rejection sampling during the fine-tuning stage to filter out scenes with object collisions, our model further reduces intersections and enhances scene quality. Additionally, we introduce a refined dataset, 3DFRONT-NC, which eliminates significant noise presented in the original dataset, 3D-FRONT. Extensive experiments on the 3D-FRONT dataset as well as our dataset demonstrate that our approach consistently outperforms the state-of-the-art methods, enhancing the realism of generated scenes, and providing a promising direction for 3D scene synthesis.
Abstract (translated)
我们提出了一种新颖的室内场景合成方法，该方法通过学习安排分解的立方体基元来表示场景中的三维对象。与传统的使用边界框确定三维物体放置和比例的方法不同，我们的方法利用立方体作为建模物体的直接且高度有效的方式。这使得在减少物体相交的同时能够生成紧凑的场景。我们称之为CasaGPT（用于立方体排列和场景组装）的方法采用自回归模型顺序安排立方体，从而生成物理上合理的场景。通过在微调阶段应用拒绝采样来过滤掉有对象碰撞的场景，我们的模型进一步减少了相交并提高了场景质量。此外，我们还引入了一个改进的数据集3DFRONT-NC，该数据集消除了原始数据集3D-FRONT中存在的大量噪声。在3D-FRONT和我们自己的数据集上的广泛实验表明，我们的方法始终优于最先进的方法，增强了生成场景的现实感，并为三维场景合成提供了有前景的方向。
URL
https://arxiv.org/abs/2504.19478
PDF
-
WORLDMEM: Long-term Consistent World Simulation with Memory
2025-04-16 17:59:30Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, Xingang PanAbstract
World simulation has gained increasing popularity due to its ability to model virtual environments and predict the consequences of actions. However, the limited temporal context window often leads to failures in maintaining long-term consistency, particularly in preserving 3D spatial consistency. In this work, we present WorldMem, a framework that enhances scene generation with a memory bank consisting of memory units that store memory frames and states (e.g., poses and timestamps). By employing a memory attention mechanism that effectively extracts relevant information from these memory frames based on their states, our method is capable of accurately reconstructing previously observed scenes, even under significant viewpoint or temporal gaps. Furthermore, by incorporating timestamps into the states, our framework not only models a static world but also captures its dynamic evolution over time, enabling both perception and interaction within the simulated world. Extensive experiments in both virtual and real scenarios validate the effectiveness of our approach.
Abstract (translated)
世界模拟由于其能够建模虚拟环境并预测行动后果而日益流行。然而，有限的时间上下文窗口经常导致在保持长期一致性方面出现问题，特别是在维持3D空间一致性上。为此，我们提出了WorldMem框架，该框架通过引入一个存储记忆单元的记忆库来增强场景生成能力，这些记忆单元存储了记忆帧和状态（如姿态和时间戳）。通过采用有效的记忆注意力机制从这些记忆帧中提取相关信息，我们的方法能够在显著的视角或时间间隔下准确重构先前观察到的场景。此外，通过在状态中加入时间戳，我们的框架不仅能够模拟静态世界，还能捕捉其随时间的变化发展过程，从而支持感知和与虚拟世界的交互。我们在虚拟和真实场景中的大量实验验证了我们这种方法的有效性。
URL
https://arxiv.org/abs/2504.12369
PDF
-
Recent Advance in 3D Object and Scene Generation: A Survey
2025-04-16 03:22:06Xiang Tang, Ruotong Li, Xiaopeng FanAbstract
In recent years, the demand for 3D content has grown exponentially with intelligent upgrading of interactive media, extended reality (XR), and Metaverse industries. In order to overcome the limitation of traditional manual modeling approaches, such as labor-intensive workflows and prolonged production cycles, revolutionary advances have been achieved through the convergence of novel 3D representation paradigms and artificial intelligence generative technologies. In this survey, we conduct a systematically review of the cutting-edge achievements in static 3D object and scene generation, as well as establish a comprehensive technical framework through systematic categorization. Specifically, we initiate our analysis with mainstream 3D object representations, followed by in-depth exploration of two principal technical pathways in object generation: data-driven supervised learning methods and deep generative model-based approaches. Regarding scene generation, we focus on three dominant paradigms: layout-guided compositional synthesis, 2D prior-based scene generation, and rule-driven modeling. Finally, we critically examine persistent challenges in 3D generation and propose potential research directions for future investigation. This survey aims to provide readers with a structured understanding of state-of-the-art 3D generation technologies while inspiring researchers to undertake more exploration in this domain.
Abstract (translated)
近年来，随着交互式媒体、扩展现实（XR）和元宇宙行业的智能化升级，对三维内容的需求呈指数级增长。为了克服传统手动建模方法的限制，如劳动密集型工作流程和延长的生产周期，通过新型三维表示范例与人工智能生成技术的融合，取得了革命性的进展。在本次综述中，我们系统地回顾了静态3D物体和场景生成领域的最新成就，并通过系统的分类建立了全面的技术框架。具体而言，我们的分析始于主流的3D对象表示形式，随后深入探讨两个主要的对象生成技术路径：数据驱动监督学习方法和基于深度生成模型的方法。对于场景生成，我们重点关注三个主导范式：布局引导组合合成、基于2D先验的场景生成以及规则驱动建模。最后，我们批判性地审视了3D生成中持续存在的挑战，并提出了未来研究方向的可能性。本次综述旨在为读者提供对最新3D生成技术结构化理解的同时，激励研究人员在此领域进行更多探索。
URL
https://arxiv.org/abs/2504.11734
PDF
-
Decoupled Diffusion Sparks Adaptive Scene Generation
2025-04-14 17:59:57Yunsong Zhou, Naisheng Ye, William Ljungbergh, Tianyu Li, Jiazhi Yang, Zetong Yang, Hongzi Zhu, Christoffer Petersson, Hongyang LiAbstract
Controllable scene generation could reduce the cost of diverse data collection substantially for autonomous driving. Prior works formulate the traffic layout generation as predictive progress, either by denoising entire sequences at once or by iteratively predicting the next frame. However, full sequence denoising hinders online reaction, while the latter's short-sighted next-frame prediction lacks precise goal-state guidance. Further, the learned model struggles to generate complex or challenging scenarios due to a large number of safe and ordinal driving behaviors from open datasets. To overcome these, we introduce Nexus, a decoupled scene generation framework that improves reactivity and goal conditioning by simulating both ordinal and challenging scenarios from fine-grained tokens with independent noise states. At the core of the decoupled pipeline is the integration of a partial noise-masking training strategy and a noise-aware schedule that ensures timely environmental updates throughout the denoising process. To complement challenging scenario generation, we collect a dataset consisting of complex corner cases. It covers 540 hours of simulated data, including high-risk interactions such as cut-in, sudden braking, and collision. Nexus achieves superior generation realism while preserving reactivity and goal orientation, with a 40% reduction in displacement error. We further demonstrate that Nexus improves closed-loop planning by 20% through data augmentation and showcase its capability in safety-critical data generation.
Abstract (translated)
可控场景生成可以大幅降低自动驾驶中多样化数据收集的成本。先前的研究将交通布局的生成视为预测性进展，要么一次去除整个序列中的噪声，要么通过迭代预测下一帧来实现。然而，一次性全序列去噪会阻碍在线反应能力，而后者仅基于下一帧的短期预测又缺乏精确的目标状态指导。此外，由于开放数据集中存在大量安全和常规驾驶行为，学习模型难以生成复杂或具有挑战性的场景。 为了克服这些问题，我们引入了Nexus框架，这是一个解耦的场景生成框架，通过模拟带有独立噪声状态的细粒度令牌，来改善反应性和目标导向性，同时可以生成正常情况及有挑战性的场景。该框架的核心在于集成部分噪声屏蔽训练策略和感知噪声的时间表安排，以确保在整个去噪过程中及时更新环境。 为了补充对具有挑战性场景的生成，我们收集了一个包含复杂边缘案例的数据集，其中包括540小时模拟数据（如切入、突然刹车和碰撞等高风险互动）。Nexus在保持反应性和目标导向的同时实现了更真实的场景生成，并将位移误差减少了40%。此外，我们还展示了通过数据增强方法来提升闭环规划的20%，并证明了其在安全关键性数据生成方面的能力。
URL
https://arxiv.org/abs/2504.10485
PDF
-
GaussVideoDreamer: 3D Scene Generation with Video Diffusion and Inconsistency-Aware Gaussian Splatting
2025-04-14 09:04:01Junlin Hao, Peiheng Wang, Haoyang Wang, Xinggong Zhang, Zongming GuoAbstract
Single-image 3D scene reconstruction presents significant challenges due to its inherently ill-posed nature and limited input constraints. Recent advances have explored two promising directions: multiview generative models that train on 3D consistent datasets but struggle with out-of-distribution generalization, and 3D scene inpainting and completion frameworks that suffer from cross-view inconsistency and suboptimal error handling, as they depend exclusively on depth data or 3D smoothness, which ultimately degrades output quality and computational performance. Building upon these approaches, we present GaussVideoDreamer, which advances generative multimedia approaches by bridging the gap between image, video, and 3D generation, integrating their strengths through two key innovations: (1) A progressive video inpainting strategy that harnesses temporal coherence for improved multiview consistency and faster convergence. (2) A 3D Gaussian Splatting consistency mask to guide the video diffusion with 3D consistent multiview evidence. Our pipeline combines three core components: a geometry-aware initialization protocol, Inconsistency-Aware Gaussian Splatting, and a progressive video inpainting strategy. Experimental results demonstrate that our approach achieves 32% higher LLaVA-IQA scores and at least 2x speedup compared to existing methods while maintaining robust performance across diverse scenes.
Abstract (translated)
单张图像的三维场景重建面临巨大的挑战，由于其本质上是一个病态问题，并且输入信息有限。最近的研究主要探索了两个有前景的方向：多视图生成模型训练在3D一致的数据集上，但难以处理出分布数据的一般化；以及依赖于深度数据或三维平滑性的3D场景修复和补全框架，在跨视角一致性方面存在不足，并且误差处理欠佳，最终导致输出质量降低及计算性能下降。基于这些方法的进展，我们提出了GaussVideoDreamer，该模型通过连接图像、视频与三维生成之间的差距，利用两个关键创新来推动生成多媒体的方法：（1）渐进式视频修复策略，借助时间一致性实现多视图一致性的改进和更快收敛。（2）3D高斯点阵一致性掩模，为视频扩散过程提供基于3D一致性多视角证据的指导。我们的流程整合了三个核心组件：几何感知初始化协议、跨一致性意识高斯点阵以及渐进式视频修复策略。实验结果表明，相较于现有的方法，我们提出的方法在LLaVA-IQA评分上提高了32%，同时至少提升了两倍的速度，并保持了在各种场景中的稳定性能。
URL
https://arxiv.org/abs/2504.10001
PDF
-
Text To 3D Object Generation For Scalable Room Assembly
2025-04-12 20:13:07Sonia Laguna, Alberto Garcia-Garcia, Marie-Julie Rakotosaona, Stylianos Moschoglou, Leonhard Helminger, Sergio Orts-EscolanoAbstract
Modern machine learning models for scene understanding, such as depth estimation and object tracking, rely on large, high-quality datasets that mimic real-world deployment scenarios. To address data scarcity, we propose an end-to-end system for synthetic data generation for scalable, high-quality, and customizable 3D indoor scenes. By integrating and adapting text-to-image and multi-view diffusion models with Neural Radiance Field-based meshing, this system generates highfidelity 3D object assets from text prompts and incorporates them into pre-defined floor plans using a rendering tool. By introducing novel loss functions and training strategies into existing methods, the system supports on-demand scene generation, aiming to alleviate the scarcity of current available data, generally manually crafted by artists. This system advances the role of synthetic data in addressing machine learning training limitations, enabling more robust and generalizable models for real-world applications.
Abstract (translated)
现代的机器学习模型，如场景理解中的深度估计和对象跟踪，依赖于大量高质量的数据集，这些数据集能够模拟实际部署环境。为了解决数据稀缺的问题，我们提出了一种端到端系统，用于生成可扩展、高质量且可定制的3D室内场景的合成数据。通过将文本到图像和多视角扩散模型与基于神经辐射场(meshing)的技术集成并适应，该系统可以从文本提示中生成高保真的3D对象资产，并利用渲染工具将其整合进预定义的平面图中。通过对现有方法引入新的损失函数和训练策略，此系统支持按需场景生成，旨在缓解目前由艺术家手工制作的数据稀缺问题。该系统推进了合成数据在解决机器学习训练限制中的作用，使得为实际应用开发出更稳健且更具泛化能力的模型成为可能。
URL
https://arxiv.org/abs/2504.09328
PDF
-
EP-Diffuser: An Efficient Diffusion Model for Traffic Scene Generation and Prediction via Polynomial Representations
2025-04-07 18:45:49Yue Yao, Mohamed-Khalil Bouzidi, Daniel Goehring, Joerg ReichardtAbstract
As the prediction horizon increases, predicting the future evolution of traffic scenes becomes increasingly difficult due to the multi-modal nature of agent motion. Most state-of-the-art (SotA) prediction models primarily focus on forecasting the most likely future. However, for the safe operation of autonomous vehicles, it is equally important to cover the distribution for plausible motion alternatives. To address this, we introduce EP-Diffuser, a novel parameter-efficient diffusion-based generative model designed to capture the distribution of possible traffic scene evolutions. Conditioned on road layout and agent history, our model acts as a predictor and generates diverse, plausible scene continuations. We benchmark EP-Diffuser against two SotA models in terms of accuracy and plausibility of predictions on the Argoverse 2 dataset. Despite its significantly smaller model size, our approach achieves both highly accurate and plausible traffic scene predictions. We further evaluate model generalization ability in an out-of-distribution (OoD) test setting using Waymo Open dataset and show superior robustness of our approach. The code and model checkpoints can be found here: this https URL.
Abstract (translated)
随着预测时间范围的增加，由于代理运动的多模态特性，对未来交通场景演化的预测难度也在增大。目前最先进的（SotA）预测模型主要集中在预测最有可能的未来情景上。然而，对于自主车辆的安全操作而言，涵盖可能的替代运动方案的概率分布同样重要。为了解决这个问题，我们引入了EP-Diffuser，这是一种新颖的参数高效扩散生成模型，旨在捕捉交通场景演化的各种可能性分布。基于道路布局和代理历史信息，我们的模型作为预测器工作，并产生多样且合理的场景延续。 我们在Argoverse 2数据集上对EP-Diffuser与两个SotA模型在预测准确性和合理性方面进行了基准测试。尽管模型大小显著较小，但我们的方法实现了高度精确且合理的交通场景预测。此外，我们使用Waymo Open数据集在外分布（OoD）测试设置中评估了模型的泛化能力，并展示了我们方法的优越鲁棒性。 代码和模型检查点可以在以下链接找到：[此URL](this https URL)。
URL
https://arxiv.org/abs/2504.05422
PDF
-
PanoDreamer: Consistent Text to 360-Degree Scene Generation
2025-04-07 14:57:01Zhexiao Xiong, Zhang Chen, Zhong Li, Yi Xu, Nathan JacobsAbstract
Automatically generating a complete 3D scene from a text description, a reference image, or both has significant applications in fields like virtual reality and gaming. However, current methods often generate low-quality textures and inconsistent 3D structures. This is especially true when extrapolating significantly beyond the field of view of the reference image. To address these challenges, we propose PanoDreamer, a novel framework for consistent, 3D scene generation with flexible text and image control. Our approach employs a large language model and a warp-refine pipeline, first generating an initial set of images and then compositing them into a 360-degree panorama. This panorama is then lifted into 3D to form an initial point cloud. We then use several approaches to generate additional images, from different viewpoints, that are consistent with the initial point cloud and expand/refine the initial point cloud. Given the resulting set of images, we utilize 3D Gaussian Splatting to create the final 3D scene, which can then be rendered from different viewpoints. Experiments demonstrate the effectiveness of PanoDreamer in generating high-quality, geometrically consistent 3D scenes.
Abstract (translated)
从文本描述、参考图像或两者结合自动生成完整的三维场景，在虚拟现实和游戏等领域具有重要的应用价值。然而，目前的方法往往生成质量较低的纹理，并且在构建不一致的三维结构时尤其如此，尤其是在超出参考图像视野范围的情况下。为了解决这些问题，我们提出了PanoDreamer，这是一种新颖的框架，用于通过灵活的文字和图像控制来生成一致性的三维场景。 我们的方法采用大型语言模型和一种扭曲-细化管道技术：首先生成一系列初始图像，然后将这些图像组合成360度全景图。接着，我们将该全景图提升至三维空间以形成初步点云。随后，我们使用几种不同的方法从不同视角生成额外的图像，并确保这些新生成的图像与初步点云保持一致，同时扩展和细化初步点云。 基于最终的一系列图像，我们利用3D高斯光束技术来创建最终的三维场景，该场景可以从多个视角进行渲染。实验结果表明，PanoDreamer在生成高质量且几何上一致的三维场景方面具有显著的效果。
URL
https://arxiv.org/abs/2504.05152
PDF
