Scene_Generation
-
IDCNet: Guided Video Diffusion for Metric-Consistent RGBD Scene Generation with Precise Camera Control
2025-08-06 07:19:16Lijuan Liu, Wenfa Li, Dongbo Zhang, Shuo Wang, Shaohui JiaoAbstract
We present IDC-Net (Image-Depth Consistency Network), a novel framework designed to generate RGB-D video sequences under explicit camera trajectory control. Unlike approaches that treat RGB and depth generation separately, IDC-Net jointly synthesizes both RGB images and corresponding depth maps within a unified geometry-aware diffusion model. The joint learning framework strengthens spatial and geometric alignment across frames, enabling more precise camera control in the generated sequences. To support the training of this camera-conditioned model and ensure high geometric fidelity, we construct a camera-image-depth consistent dataset with metric-aligned RGB videos, depth maps, and accurate camera poses, which provides precise geometric supervision with notably improved inter-frame geometric consistency. Moreover, we introduce a geometry-aware transformer block that enables fine-grained camera control, enhancing control over the generated sequences. Extensive experiments show that IDC-Net achieves improvements over state-of-the-art approaches in both visual quality and geometric consistency of generated scene sequences. Notably, the generated RGB-D sequences can be directly feed for downstream 3D Scene reconstruction tasks without extra post-processing steps, showcasing the practical benefits of our joint learning framework. See more at this https URL.
Abstract (translated)
我们提出了IDC-Net（图像深度一致性网络），这是一种新颖的框架，旨在根据明确的相机轨迹控制生成RGB-D视频序列。与将RGB和深度生成分开处理的方法不同，IDC-Net在一个统一的几何感知扩散模型中联合合成RGB图像及其对应的深度图，从而加强了帧间的空间和几何对齐，使得在生成序列中可以实现更精确的相机控制。 为了支持该条件化相机模型的训练，并确保高几何保真度，我们构建了一个包含度量一致的RGB视频、深度图及准确相机姿态的相机-图像-深度一致性数据集。该数据集提供了精准的几何监督，并且显著提高了帧间几何一致性。此外，我们还引入了一种具有几何感知能力的变压器模块，使细粒度的相机控制成为可能，从而增强对生成序列的控制。 大量的实验表明，在视觉质量和生成场景序列中的几何一致性的方面，IDC-Net相比最先进的方法实现了改进。值得注意的是，生成的RGB-D序列可以直接用于下游3D场景重建任务而无需额外的后处理步骤，展示了我们联合学习框架的实际益处。有关更多信息，请访问此链接：[https URL]（请注意这里的URL应替换为实际提供的链接）。
URL
https://arxiv.org/abs/2508.04147
PDF
-
La La LiDAR: Large-Scale Layout Generation from LiDAR Data
2025-08-05 17:59:55Youquan Liu, Lingdong Kong, Weidong Yang, Xin Li, Ao Liang, Runnan Chen, Ben Fei, Tongliang LiuAbstract
Controllable generation of realistic LiDAR scenes is crucial for applications such as autonomous driving and robotics. While recent diffusion-based models achieve high-fidelity LiDAR generation, they lack explicit control over foreground objects and spatial relationships, limiting their usefulness for scenario simulation and safety validation. To address these limitations, we propose Large-scale Layout-guided LiDAR generation model ("La La LiDAR"), a novel layout-guided generative framework that introduces semantic-enhanced scene graph diffusion with relation-aware contextual conditioning for structured LiDAR layout generation, followed by foreground-aware control injection for complete scene generation. This enables customizable control over object placement while ensuring spatial and semantic consistency. To support our structured LiDAR generation, we introduce Waymo-SG and nuScenes-SG, two large-scale LiDAR scene graph datasets, along with new evaluation metrics for layout synthesis. Extensive experiments demonstrate that La La LiDAR achieves state-of-the-art performance in both LiDAR generation and downstream perception tasks, establishing a new benchmark for controllable 3D scene generation.
Abstract (translated)
可控生成逼真的激光雷达场景对于自动驾驶和机器人技术等应用至关重要。尽管最近基于扩散模型的方案在高保真激光雷达生成方面取得了成功，但这些模型缺乏对前景物体及其空间关系的显式控制，这限制了它们在场景模拟和安全验证中的实用性。为了解决这些问题，我们提出了一种新的布局引导生成框架——大规模布局引导激光雷达生成模型（"La La LiDAR"），该模型引入了语义增强的场景图扩散，并结合了感知上下文关系的情境条件，用于结构化的激光雷达布局生成。随后通过前景感知控制注入来完成完整场景的生成，从而在确保空间和语义一致性的同时实现对象放置的自定义化控制。 为了支持我们的结构化激光雷达生成工作，我们还引入了Waymo-SG和nuScenes-SG两个大规模激光雷达场景图数据集，并提出了新的布局合成评估指标。大量的实验表明，La La LiDAR在激光雷达生成以及下游感知任务中均达到了最先进的性能水平，为可控的3D场景生成设定了一个新的基准标准。
URL
https://arxiv.org/abs/2508.03691
PDF
-
Graph-Guided Dual-Level Augmentation for 3D Scene Segmentation
2025-07-30 13:25:36Hongbin Lin, Yifan Jiang, Juangui Xu, Jesse Jiaxi Xu, Yi Lu, Zhengyu Hu, Ying-Cong Chen, Hao WangAbstract
3D point cloud segmentation aims to assign semantic labels to individual points in a scene for fine-grained spatial understanding. Existing methods typically adopt data augmentation to alleviate the burden of large-scale annotation. However, most augmentation strategies only focus on local transformations or semantic recomposition, lacking the consideration of global structural dependencies within scenes. To address this limitation, we propose a graph-guided data augmentation framework with dual-level constraints for realistic 3D scene synthesis. Our method learns object relationship statistics from real-world data to construct guiding graphs for scene generation. Local-level constraints enforce geometric plausibility and semantic consistency between objects, while global-level constraints maintain the topological structure of the scene by aligning the generated layout with the guiding graph. Extensive experiments on indoor and outdoor datasets demonstrate that our framework generates diverse and high-quality augmented scenes, leading to consistent improvements in point cloud segmentation performance across various models.
Abstract (translated)
3D点云分割的目标是为场景中的单个点分配语义标签，以实现精细的三维空间理解。现有的方法通常采用数据增强来减轻大规模标注的工作量。然而，大多数增强策略仅关注局部变换或语义重组，缺乏对场景内全局结构依赖性的考虑。为了克服这一限制，我们提出了一种基于图引导的数据增强框架，该框架具有双层约束机制，用于生成现实的三维场景。我们的方法从真实世界数据中学习对象关系统计信息，并以此构建指导图以进行场景生成。局部级别的约束强制执行几何合理性和对象之间的语义一致性，而全局级别的约束通过使生成的布局与指导图对齐来保持场景的拓扑结构。在室内和室外数据集上进行了广泛的实验，结果表明我们的框架能够生成多样化且高质量的数据增强场景，并能在各种模型中显著提升点云分割性能。
URL
https://arxiv.org/abs/2507.22668
PDF
-
TopoLiDM: Topology-Aware LiDAR Diffusion Models for Interpretable and Realistic LiDAR Point Cloud Generation
2025-07-30 08:02:42Jiuming Liu, Zheng Huang, Mengmeng Liu, Tianchen Deng, Francesco Nex, Hao Cheng, Hesheng WangAbstract
LiDAR scene generation is critical for mitigating real-world LiDAR data collection costs and enhancing the robustness of downstream perception tasks in autonomous driving. However, existing methods commonly struggle to capture geometric realism and global topological consistency. Recent LiDAR Diffusion Models (LiDMs) predominantly embed LiDAR points into the latent space for improved generation efficiency, which limits their interpretable ability to model detailed geometric structures and preserve global topological consistency. To address these challenges, we propose TopoLiDM, a novel framework that integrates graph neural networks (GNNs) with diffusion models under topological regularization for high-fidelity LiDAR generation. Our approach first trains a topological-preserving VAE to extract latent graph representations by graph construction and multiple graph convolutional layers. Then we freeze the VAE and generate novel latent topological graphs through the latent diffusion models. We also introduce 0-dimensional persistent homology (PH) constraints, ensuring the generated LiDAR scenes adhere to real-world global topological structures. Extensive experiments on the KITTI-360 dataset demonstrate TopoLiDM's superiority over state-of-the-art methods, achieving improvements of 22.6% lower Frechet Range Image Distance (FRID) and 9.2% lower Minimum Matching Distance (MMD). Notably, our model also enables fast generation speed with an average inference time of 1.68 samples/s, showcasing its scalability for real-world applications. We will release the related codes at this https URL.
Abstract (translated)
LiDAR场景生成对于降低真实世界中的激光雷达数据采集成本以及增强自动驾驶下游感知任务的鲁棒性至关重要。然而，现有方法通常难以捕捉几何真实性及全局拓扑一致性。最近提出的激光雷达扩散模型（LiDMs）主要通过将激光雷达点嵌入到潜在空间来提高生成效率，这限制了它们在建模详细几何结构和保持全局拓扑一致方面的解释能力。为了解决这些挑战，我们提出了一种新框架TopoLiDM，该框架整合了图神经网络（GNNs）与扩散模型，并通过拓扑正则化实现高保真度的激光雷达生成。 我们的方法首先训练一个保持拓扑结构的变分自编码器（VAE），通过构建图和多个图卷积层来提取潜在图表示。然后，我们将该VAE冻结并利用潜扩散模型生成新的潜在拓扑图。此外，我们还引入了零维持久同调（PH）约束，确保所生成的激光雷达场景符合真实世界的全局拓扑结构。 在KITTI-360数据集上进行的大量实验表明，TopoLiDM相比现有最先进方法有明显优势，在Frechet范围图像距离（FRID）和最小匹配距离（MMD）方面分别取得了22.6%和9.2%的改进。值得注意的是，我们的模型还具备快速生成速度，平均推断时间为每秒1.68个样本，展示了其在实际应用中的可扩展性。 相关代码将在以下网址发布：[链接] (请提供URL)。
URL
https://arxiv.org/abs/2507.22454
PDF
-
ScenePainter: Semantically Consistent Perpetual 3D Scene Generation with Concept Relation Alignment
2025-07-25 08:21:12Chong Xia, Shengjun Zhang, Fangfu Liu, Chang Liu, Khodchaphun Hirunyaratsameewong, Yueqi DuanAbstract
Perpetual 3D scene generation aims to produce long-range and coherent 3D view sequences, which is applicable for long-term video synthesis and 3D scene reconstruction. Existing methods follow a "navigate-and-imagine" fashion and rely on outpainting for successive view expansion. However, the generated view sequences suffer from semantic drift issue derived from the accumulated deviation of the outpainting module. To tackle this challenge, we propose ScenePainter, a new framework for semantically consistent 3D scene generation, which aligns the outpainter's scene-specific prior with the comprehension of the current scene. To be specific, we introduce a hierarchical graph structure dubbed SceneConceptGraph to construct relations among multi-level scene concepts, which directs the outpainter for consistent novel views and can be dynamically refined to enhance diversity. Extensive experiments demonstrate that our framework overcomes the semantic drift issue and generates more consistent and immersive 3D view sequences. Project Page: this https URL.
Abstract (translated)
持续的3D场景生成旨在生成长跨度且连贯的3D视图序列，这适用于长期视频合成和3D场景重建。现有方法遵循“导航与想象”的方式，并依赖于扩展（outpainting）来实现连续视图的扩展。然而，所生成的视图序列会因累积误差导致语义漂移问题。为解决这一挑战，我们提出了ScenePainter，这是一个新的框架，用于生成语义上一致的3D场景，它通过将扩展器的特定于场景的先验知识与当前场景的理解进行对齐来实现这一目标。 具体而言，我们引入了一种层次化的图结构，名为SceneConceptGraph，用以构建多级场景概念之间的关系。这种图结构可以指导生成器产生一致的新视图，并且可以通过动态细化增强多样性。 广泛的实验表明，我们的框架能够克服语义漂移问题，并生成更为一致和沉浸式的3D视图序列。 项目页面：[此链接](this https URL)。
URL
https://arxiv.org/abs/2507.19058
PDF
-
Enhancing Scene Transition Awareness in Video Generation via Post-Training
2025-07-24 02:50:26Hanwen Shen, Jiajie Lu, Yupeng Cao, Xiaonan YangAbstract
Recent advances in AI-generated video have shown strong performance on \emph{text-to-video} tasks, particularly for short clips depicting a single scene. However, current models struggle to generate longer videos with coherent scene transitions, primarily because they cannot infer when a transition is needed from the prompt. Most open-source models are trained on datasets consisting of single-scene video clips, which limits their capacity to learn and respond to prompts requiring multiple scenes. Developing scene transition awareness is essential for multi-scene generation, as it allows models to identify and segment videos into distinct clips by accurately detecting transitions. To address this, we propose the \textbf{Transition-Aware Video} (TAV) dataset, which consists of preprocessed video clips with multiple scene transitions. Our experiment shows that post-training on the \textbf{TAV} dataset improves prompt-based scene transition understanding, narrows the gap between required and generated scenes, and maintains image quality.
Abstract (translated)
最近在AI生成视频领域的进展显示，在将文本转换为短视频的任务中表现出色，尤其是对于描绘单一场景的短片段。然而，当前的模型难以生成包含连贯场景过渡的较长视频，因为它们无法从提示中推断出何时需要进行场景切换。大多数开源模型都是基于由单个场景组成的视频剪辑的数据集训练出来的，这限制了它们对要求多场景响应的提示的学习和应对能力。为了提高多场景生成的效果，发展场景转换意识至关重要，它使模型能够准确地检测并分割视频中的不同片段。为了解决这个问题，我们提出了**过渡感知视频（Transition-Aware Video, TAV）数据集**，该数据集包含具有多个场景转换的预处理视频剪辑。我们的实验表明，在TAV数据集上进行后期训练可以改善基于提示的场景转换理解能力，缩小所需场景与生成场景之间的差距，并保持图像质量。
URL
https://arxiv.org/abs/2507.18046
PDF
-
Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimization
2025-07-20 06:59:42Xiang Tang, Ruotong Li, Xiaopeng FanAbstract
In recent years, 3D generation has made great strides in both academia and industry. However, generating 3D scenes from a single RGB image remains a significant challenge, as current approaches often struggle to ensure both object generation quality and scene coherence in multi-object scenarios. To overcome these limitations, we propose a novel three-stage framework for 3D scene generation with explicit geometric representations and high-quality textural details via single image-guided model generation and spatial layout optimization. Our method begins with an image instance segmentation and inpainting phase, which recovers missing details of occluded objects in the input images, thereby achieving complete generation of foreground 3D assets. Subsequently, our approach captures the spatial geometry of reference image by constructing pseudo-stereo viewpoint for camera parameter estimation and scene depth inference, while employing a model selection strategy to ensure optimal alignment between the 3D assets generated in the previous step and the input. Finally, through model parameterization and minimization of the Chamfer distance between point clouds in 3D and 2D space, our approach optimizes layout parameters to produce an explicit 3D scene representation that maintains precise alignment with input guidance image. Extensive experiments on multi-object scene image sets have demonstrated that our approach not only outperforms state-of-the-art methods in terms of geometric accuracy and texture fidelity of individual generated 3D models, but also has significant advantages in scene layout synthesis.
Abstract (translated)
近年来，3D生成技术在学术界和工业界都取得了重大进展。然而，从单张RGB图像生成3D场景仍然是一个巨大的挑战，因为现有方法常常难以同时保证多对象场景中的物体生成质量和场景一致性。为了克服这些限制，我们提出了一种新颖的三阶段框架，用于通过单一图像引导模型生成和空间布局优化来创建具有显式几何表示和高质量纹理细节的3D场景。 我们的方法首先进行图像实例分割与修复阶段，在此过程中恢复输入图片中被遮挡物体缺失的详细信息，从而实现前景3D资产的完整生成。随后，我们通过构建伪立体视角来进行相机参数估计和场景深度推断，以此捕捉参考图的空间几何结构，并采用模型选择策略确保先前步骤产生的3D资产与输入图像的最佳对齐。最后，通过对模型进行参数化处理并最小化3D和2D空间中点云之间的Chamfer距离，我们的方法优化布局参数，从而生成一个显式的3D场景表示，保持精确地与指导输入图一致。 在多对象场景图像集上的广泛实验表明，我们所提出的方法不仅在单个生成的3D模型的几何准确性和纹理保真度方面超越了最先进的技术，还在场景布局合成方面具有显著优势。
URL
https://arxiv.org/abs/2507.14841
PDF
-
DreamScene: 3D Gaussian-based End-to-end Text-to-3D Scene Generation
2025-07-18 14:45:54Haoran Li, Yuli Tian, Kun Lan, Yong Liao, Lin Wang, Pan Hui, Peng Yuan ZhouAbstract
Generating 3D scenes from natural language holds great promise for applications in gaming, film, and design. However, existing methods struggle with automation, 3D consistency, and fine-grained control. We present DreamScene, an end-to-end framework for high-quality and editable 3D scene generation from text or dialogue. DreamScene begins with a scene planning module, where a GPT-4 agent infers object semantics and spatial constraints to construct a hybrid graph. A graph-based placement algorithm then produces a structured, collision-free layout. Based on this layout, Formation Pattern Sampling (FPS) generates object geometry using multi-timestep sampling and reconstructive optimization, enabling fast and realistic synthesis. To ensure global consistent, DreamScene employs a progressive camera sampling strategy tailored to both indoor and outdoor settings. Finally, the system supports fine-grained scene editing, including object movement, appearance changes, and 4D dynamic motion. Experiments demonstrate that DreamScene surpasses prior methods in quality, consistency, and flexibility, offering a practical solution for open-domain 3D content creation. Code and demos are available at this https URL.
Abstract (translated)
从自然语言生成三维场景在游戏、电影和设计等领域具有巨大的潜力。然而，现有的方法在自动化、三维一致性以及精细控制方面存在挑战。我们提出了DreamScene，这是一个端到端的框架，用于根据文本或对话生成高质量且可编辑的三维场景。DreamScene 以场景规划模块开始，其中GPT-4代理推断出对象语义和空间约束，并构建了一个混合图。随后，基于图的放置算法产生一个结构化且无碰撞的布局。在此布局的基础上，形成模式采样（FPS）通过多时间步采样和重构优化生成物体几何形状，从而实现快速而逼真的合成。为了确保全局一致性，DreamScene 采用了一种针对室内和室外场景量身定制的渐进式相机采样策略。最后，该系统支持精细的场景编辑功能，包括对象移动、外观变化以及四维动态运动。实验表明，DreamScene 在质量、一致性和灵活性方面超越了先前的方法，为开放领域的三维内容创作提供了一个实用解决方案。代码和演示可在[此链接](https://this-url.com)获取。
URL
https://arxiv.org/abs/2507.13985
PDF
-
World Model-Based End-to-End Scene Generation for Accident Anticipation in Autonomous Driving
2025-07-17 03:34:54Yanchen Guan, Haicheng Liao, Chengyue Wang, Xingcheng Liu, Jiaxun Zhang, Zhenning LiAbstract
Reliable anticipation of traffic accidents is essential for advancing autonomous driving systems. However, this objective is limited by two fundamental challenges: the scarcity of diverse, high-quality training data and the frequent absence of crucial object-level cues due to environmental disruptions or sensor deficiencies. To tackle these issues, we propose a comprehensive framework combining generative scene augmentation with adaptive temporal reasoning. Specifically, we develop a video generation pipeline that utilizes a world model guided by domain-informed prompts to create high-resolution, statistically consistent driving scenarios, particularly enriching the coverage of edge cases and complex interactions. In parallel, we construct a dynamic prediction model that encodes spatio-temporal relationships through strengthened graph convolutions and dilated temporal operators, effectively addressing data incompleteness and transient visual noise. Furthermore, we release a new benchmark dataset designed to better capture diverse real-world driving risks. Extensive experiments on public and newly released datasets confirm that our framework enhances both the accuracy and lead time of accident anticipation, offering a robust solution to current data and modeling limitations in safety-critical autonomous driving applications.
Abstract (translated)
可靠地预测交通事故对于推进自动驾驶系统的发展至关重要。然而，这一目标受到了两个基本挑战的限制：高质量多样化训练数据的稀缺性以及由于环境干扰或传感器缺陷而导致的重要对象级线索经常缺失的问题。为了解决这些问题，我们提出了一种结合生成场景增强和自适应时间推理的全面框架。 具体而言，我们开发了一个视频生成管道，该管道利用由领域信息提示指导的世界模型来创建高分辨率且统计上一致的驾驶场景，并特别丰富了边缘案例和复杂交互的情况。同时，我们构建了一个动态预测模型，通过加强的图卷积和膨胀的时间操作符编码空间-时间关系，有效地解决了数据不完整性和短暂视觉噪声的问题。 此外，我们发布了一套新的基准数据集，旨在更好地捕捉各种现实世界驾驶风险。在公开数据集和新发布的数据集上的广泛实验表明，我们的框架增强了事故预测的准确性和提前期，为安全关键性的自动驾驶应用提供了应对当前数据和建模限制的有效解决方案。
URL
https://arxiv.org/abs/2507.12762
PDF
-
Foundation Model Driven Robotics: A Comprehensive Review
2025-07-14 09:13:07Muhammad Tayyab Khan, Ammar WaheedAbstract
The rapid emergence of foundation models, particularly Large Language Models (LLMs) and Vision-Language Models (VLMs), has introduced a transformative paradigm in robotics. These models offer powerful capabilities in semantic understanding, high-level reasoning, and cross-modal generalization, enabling significant advances in perception, planning, control, and human-robot interaction. This critical review provides a structured synthesis of recent developments, categorizing applications across simulation-driven design, open-world execution, sim-to-real transfer, and adaptable robotics. Unlike existing surveys that emphasize isolated capabilities, this work highlights integrated, system-level strategies and evaluates their practical feasibility in real-world environments. Key enabling trends such as procedural scene generation, policy generalization, and multimodal reasoning are discussed alongside core bottlenecks, including limited embodiment, lack of multimodal data, safety risks, and computational constraints. Through this lens, this paper identifies both the architectural strengths and critical limitations of foundation model-based robotics, highlighting open challenges in real-time operation, grounding, resilience, and trust. The review concludes with a roadmap for future research aimed at bridging semantic reasoning and physical intelligence through more robust, interpretable, and embodied models.
Abstract (translated)
基础模型（尤其是大型语言模型和视觉-语言模型）的迅速发展为机器人技术带来了变革性的范式。这些模型提供了强大的语义理解、高级推理和跨模态泛化能力，从而在感知、规划、控制以及人机交互方面取得了重大进展。这篇批判性综述系统地总结了最近的发展，并根据模拟驱动设计、开放环境执行、从模拟到现实的迁移以及适应性机器人等应用对这些发展进行了分类。与强调孤立功能的现有调查不同，这项工作突出了集成的、系统级策略，并评估了它们在实际环境中的可行性。 文中还讨论了一些关键的技术趋势，如程序化场景生成、政策泛化和多模态推理，同时指出了核心瓶颈问题，包括有限的身体化能力、缺乏多模态数据、安全风险以及计算限制。通过这一视角，本文不仅识别出基于基础模型的机器人架构的优势和局限性，还强调了实时操作、实体化、弹性和信任方面的开放挑战。 综述最后提出了一条未来研究路线图，旨在通过更强大、可解释且具身体化的模型来弥合语义推理与物理智能之间的差距。
URL
https://arxiv.org/abs/2507.10087
PDF
-
$I^{2}$-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting
2025-07-12 05:14:39Zhimin Liao, Ping Wei, Ruijie Zhang, Shuaijia Chen, Haoxuan Wang, Ziyang RenAbstract
Forecasting the evolution of 3D scenes and generating unseen scenarios via occupancy-based world models offers substantial potential for addressing corner cases in autonomous driving systems. While tokenization has revolutionized image and video generation, efficiently tokenizing complex 3D scenes remains a critical challenge for 3D world models. To address this, we propose $I^{2}$-World, an efficient framework for 4D occupancy forecasting. Our method decouples scene tokenization into intra-scene and inter-scene tokenizers. The intra-scene tokenizer employs a multi-scale residual quantization strategy to hierarchically compress 3D scenes while preserving spatial details. The inter-scene tokenizer residually aggregates temporal dependencies across timesteps. This dual design preserves the compactness of 3D tokenizers while retaining the dynamic expressiveness of 4D tokenizers. Unlike decoder-only GPT-style autoregressive models, $I^{2}$-World adopts an encoder-decoder architecture. The encoder aggregates spatial context from the current scene and predicts a transformation matrix to enable high-level control over scene generation. The decoder, conditioned on this matrix and historical tokens, ensures temporal consistency during generation. Experiments demonstrate that $I^{2}$-World achieves state-of-the-art performance, outperforming existing methods by 25.1\% in mIoU and 36.9\% in IoU for 4D occupancy forecasting while exhibiting exceptional computational efficiency: it requires merely 2.9 GB of training memory and achieves real-time inference at 37.0 FPS. Our code is available on this https URL.
Abstract (translated)
基于占用的模型预测三维场景的发展并生成未见过的情景，为解决自动驾驶系统中的边缘情况提供了巨大的潜力。虽然标记化技术在图像和视频生成领域带来了革命性的变革，但高效地对复杂的三维场景进行标记仍然是三维世界模型面临的一个关键挑战。为此，我们提出了$I^{2}$-World，这是一个用于四维占用预测的高效框架。我们的方法将场景标记分解为内部场景标记器（intra-scene tokenizer）和跨场景标记器（inter-scene tokenizer）。 **内部场景标记器**采用多尺度残差量化策略来分层压缩三维场景，同时保留空间细节；而**跨场景标记器**则通过逐时间步的方式聚合时序依赖性。这种双管齐下的设计在保持3D标记器的紧凑性的同时，还保留了4D标记器的动态表达能力。 与解码器独占的GPT风格的自回归模型不同，$I^{2}$-World采用了一种编码器-解码器架构。编码器从当前场景中聚合空间上下文，并预测一个转换矩阵以实现对场景生成的高级控制；而解码器则根据该矩阵和历史标记，在生成过程中确保时间一致性。 实验表明，$I^{2}$-World在四维占用预测方面取得了最先进的性能，与现有方法相比，其mIoU（平均交并比）提高了25.1%，IoU（交并比）提高了36.9%。此外，它还展示了卓越的计算效率：仅需2.9 GB的训练内存，并在实时推理中达到每秒37.0帧的速度。 我们的代码可以在以下链接获取: [请在此处插入URL]。
URL
https://arxiv.org/abs/2507.09144
PDF
-
Acquiring and Adapting Priors for Novel Tasks via Neural Meta-Architectures
2025-07-07 18:48:09Sudarshan BabuAbstract
The ability to transfer knowledge from prior experiences to novel tasks stands as a pivotal capability of intelligent agents, including both humans and computational models. This principle forms the basis of transfer learning, where large pre-trained neural networks are fine-tuned to adapt to downstream tasks. Transfer learning has demonstrated tremendous success, both in terms of task adaptation speed and performance. However there are several domains where, due to lack of data, training such large pre-trained models or foundational models is not a possibility - computational chemistry, computational immunology, and medical imaging are examples. To address these challenges, our work focuses on designing architectures to enable efficient acquisition of priors when large amounts of data are unavailable. In particular, we demonstrate that we can use neural memory to enable adaptation on non-stationary distributions with only a few samples. Then we demonstrate that our hypernetwork designs (a network that generates another network) can acquire more generalizable priors than standard networks when trained with Model Agnostic Meta-Learning (MAML). Subsequently, we apply hypernetworks to 3D scene generation, demonstrating that they can acquire priors efficiently on just a handful of training scenes, thereby leading to faster text-to-3D generation. We then extend our hypernetwork framework to perform 3D segmentation on novel scenes with limited data by efficiently transferring priors from earlier viewed scenes. Finally, we repurpose an existing molecular generative method as a pre-training framework that facilitates improved molecular property prediction, addressing critical challenges in computational immunology
Abstract (translated)
将先前经验中的知识转移到新任务的能力是智能代理（包括人类和计算模型）的一个关键能力。这一原则构成了迁移学习的基础，在这种学习中，大型预训练神经网络被微调以适应下游任务。迁移学习在任务适应速度和性能方面都取得了巨大的成功。然而，在某些领域，由于数据缺乏，训练如此大规模的预训练模型或基础模型是不可能的——计算化学、计算免疫学和医学成像就是例子。为了解决这些挑战，我们的工作重点是设计架构，以便在大量数据不可用的情况下高效地获取先验知识。特别地，我们展示了如何使用神经记忆来适应非平稳分布，并且仅需少量样本即可实现这一目标。然后，我们证明了当使用模型无关元学习（MAML）训练时，我们的超网络设计（生成另一个网络的网络）可以比标准网络获得更通用的先验知识。随后，我们将超网络应用于三维场景生成，展示了它们只需少量训练场景就能高效地获取先验知识，从而加快从文本到3D生成的速度。最后，我们重新利用现有的分子生成方法作为预训练框架，以改善分子性质预测，解决了计算免疫学中的关键挑战。
URL
https://arxiv.org/abs/2507.10446
PDF
-
Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusion
2025-07-06 14:30:08Tongyan Hua, Lutao Jiang, Ying-Cong Chen, Wufan ZhaoAbstract
Recent advancements in generative models have enabled 3D urban scene generation from satellite imagery, unlocking promising applications in gaming, digital twins, and beyond. However, most existing methods rely heavily on neural rendering techniques, which hinder their ability to produce detailed 3D structures on a broader scale, largely due to the inherent structural ambiguity derived from relatively limited 2D observations. To address this challenge, we propose Sat2City, a novel framework that synergizes the representational capacity of sparse voxel grids with latent diffusion models, tailored specifically for our novel 3D city dataset. Our approach is enabled by three key components: (1) A cascaded latent diffusion framework that progressively recovers 3D city structures from satellite imagery, (2) a Re-Hash operation at its Variational Autoencoder (VAE) bottleneck to compute multi-scale feature grids for stable appearance optimization and (3) an inverse sampling strategy enabling implicit supervision for smooth appearance this http URL overcome the challenge of collecting real-world city-scale 3D models with high-quality geometry and appearance, we introduce a dataset of synthesized large-scale 3D cities paired with satellite-view height maps. Validated on this dataset, our framework generates detailed 3D structures from a single satellite image, achieving superior fidelity compared to existing city generation models.
Abstract (translated)
最近在生成模型领域的进展使得从卫星图像生成三维城市场景成为可能，这为游戏、数字孪生等领域开启了新的应用前景。然而，大多数现有方法严重依赖于神经渲染技术，这种技术限制了它们在更广泛范围内产生详细三维结构的能力，主要是由于相对有限的二维观测导致的基本结构模糊性。为了应对这一挑战，我们提出了Sat2City，这是一种新颖的框架，它将稀疏体素网格的表现能力与潜在扩散模型相结合，专为我们的新型3D城市数据集设计。 该方法由三个关键组件组成： 1. 一个级联式潜在扩散框架，逐步从卫星图像中恢复三维城市结构。 2. 在其变分自编码器（VAE）瓶颈处进行的Re-Hash操作，用于计算多尺度特征网格以实现稳定的外观优化。 3. 一种反向采样策略，通过隐性监督使外观平滑化。 为了克服收集具有高质量几何和外观的真实世界城市规模三维模型这一挑战，我们引入了一个由合成大规模3D城市及其卫星视图高度地图组成的新型数据集。在该数据集上验证后，我们的框架能够从单张卫星图像生成详细的三维结构，并且与现有城市生成模型相比，在保真度方面表现更佳。
URL
https://arxiv.org/abs/2507.04403
PDF
-
Voyaging into Unbounded Dynamic Scenes from a Single View
2025-07-05 22:49:25Fengrui Tian, Tianjiao Ding, Jinqi Luo, Hancheng Min, Ren\'e VidalAbstract
This paper studies the problem of generating an unbounded dynamic scene from a single view, which has wide applications in augmented/virtual reality and robotics. Since the scene is changing over time, different generated views need to be consistent with the underlying 3D motions. While previous works learn such consistency by training from multiple views, the generated scene regions are bounded to be close to the training views with limited camera movements. To address this issue, we propose DynamicVoyager that reformulates the dynamic scene generation as a scene outpainting process for new dynamic content. As 2D outpainting models can hardly generate 3D consistent motions from only 2D pixels at a single view, we consider pixels as rays to enrich the pixel input with the ray context, so that the 3D motion consistency can be learned from the ray information. More specifically, we first map the single-view video input to a dynamic point cloud with the estimated video depths. Then we render the partial video at a novel view and outpaint the video with ray contexts from the point cloud to generate 3D consistent motions. We employ the outpainted video to update the point cloud, which is used for scene outpainting from future novel views. Experiments show that our model is able to generate unbounded scenes with consistent motions along fly-through cameras, and the generated contents can be controlled with scene prompts.
Abstract (translated)
本文研究了从单视图生成无边界动态场景的问题，该问题在增强/虚拟现实和机器人技术中具有广泛的应用。由于场景随时间变化，不同生成的视图需要与底层3D运动保持一致。虽然先前的工作通过多视角训练来学习这种一致性，但生成的场景区域被限制在靠近训练视图且相机移动有限的位置上。为了解决这个问题，我们提出了DynamicVoyager，它将动态场景生成重新表述为新动态内容的场景外推过程（scene outpainting）。由于2D外推模型难以仅从单视图的2D像素中生成3D一致的运动，我们将像素视为射线，并通过添加射线上下文来丰富像素输入，从而可以从射线信息中学到3D运动一致性。具体来说，我们首先将单视角视频输入映射为动态点云，使用估计的视频深度信息进行映射。然后，在新视图中渲染部分视频，并利用点云中的射线上下文外推视频以生成一致的3D运动。我们采用外推后的视频来更新点云，并用于从未来的新的视图中进行场景外推。实验表明，我们的模型能够生成沿飞行穿越相机路径具有连续动作的无边界场景，并且可以使用场景提示控制生成的内容。
URL
https://arxiv.org/abs/2507.04183
PDF
-
BEV-VAE: Multi-view Image Generation with Spatial Consistency for Autonomous Driving
2025-07-01 12:10:11Zeming Chen, Hang ZhaoAbstract
Multi-view image generation in autonomous driving demands consistent 3D scene understanding across camera views. Most existing methods treat this problem as a 2D image set generation task, lacking explicit 3D modeling. However, we argue that a structured representation is crucial for scene generation, especially for autonomous driving applications. This paper proposes BEV-VAE for consistent and controllable view synthesis. BEV-VAE first trains a multi-view image variational autoencoder for a compact and unified BEV latent space and then generates the scene with a latent diffusion transformer. BEV-VAE supports arbitrary view generation given camera configurations, and optionally 3D layouts. Experiments on nuScenes and Argoverse 2 (AV2) show strong performance in both 3D consistent reconstruction and generation. The code is available at: this https URL.
Abstract (translated)
在自动驾驶中，多视角图像生成要求不同相机视图之间具有连贯的三维场景理解。大多数现有的方法将这个问题视为二维图像集生成任务，并没有明确地进行三维建模。然而，我们认为一个结构化的表示对于场景生成至关重要，特别是在自动驾驶应用中。本文提出了BEV-VAE（Bird's Eye View Variational Autoencoder），以实现一致且可控的视图合成。BEV-VAE首先训练一个多视角图像变分自编码器，以形成紧凑而统一的鸟瞰图隐式空间，然后使用一个隐式扩散变压器来生成场景。BEV-VAE支持给定相机配置以及可选的三维布局的情况下进行任意视图生成。在nuScenes和Argoverse 2（AV2）数据集上的实验表明，在3D一致重建和生成方面具有出色表现。代码可在以下链接获取：this https URL.
URL
https://arxiv.org/abs/2507.00707
PDF
-
InfGen: Scenario Generation as Next Token Group Prediction
2025-06-29 16:18:32Zhenghao Peng, Yuxin Liu, Bolei ZhouAbstract
Realistic and interactive traffic simulation is essential for training and evaluating autonomous driving systems. However, most existing data-driven simulation methods rely on static initialization or log-replay data, limiting their ability to model dynamic, long-horizon scenarios with evolving agent populations. We propose InfGen, a scenario generation framework that outputs agent states and trajectories in an autoregressive manner. InfGen represents the entire scene as a sequence of tokens, including traffic light signals, agent states, and motion vectors, and uses a transformer model to simulate traffic over time. This design enables InfGen to continuously insert new agents into traffic, supporting infinite scene generation. Experiments demonstrate that InfGen produces realistic, diverse, and adaptive traffic behaviors. Furthermore, reinforcement learning policies trained in InfGen-generated scenarios achieve superior robustness and generalization, validating its utility as a high-fidelity simulation environment for autonomous driving. More information is available at this https URL.
Abstract (translated)
现实而互动的交通模拟对于自动驾驶系统的训练和评估至关重要。然而，大多数现有的数据驱动仿真方法依赖于静态初始化或日志回放数据，这限制了它们在长时间范围内建模动态变化且不断发展的代理群体的能力。我们提出了InfGen这一场景生成框架，它能够以自回归的方式输出代理的状态和轨迹。InfGen将整个场景表示为一个包括交通信号灯、代理状态和运动向量的标记序列，并采用Transformer模型来模拟随着时间推移的变化中的交通状况。这种设计使得InfGen能够在交通中持续插入新的代理，支持无限场景生成。实验表明，InfGen能够产生现实性强、多样化且适应性好的交通行为。此外，在InfGen生成的情景中训练的强化学习策略在鲁棒性和泛化能力上表现更佳，验证了它作为自动驾驶高保真模拟环境的有效性。 更多信息可在此网址获取：[请在此处插入链接]
URL
https://arxiv.org/abs/2506.23316
PDF
-
RoomCraft: Controllable and Complete 3D Indoor Scene Generation
2025-06-27 15:03:17Mengqi Zhou, Xipeng Wang, Yuxi Wang, Zhaoxiang ZhangAbstract
Generating realistic 3D indoor scenes from user inputs remains a challenging problem in computer vision and graphics, requiring careful balance of geometric consistency, spatial relationships, and visual realism. While neural generation methods often produce repetitive elements due to limited global spatial reasoning, procedural approaches can leverage constraints for controllable generation but struggle with multi-constraint scenarios. When constraints become numerous, object collisions frequently occur, forcing the removal of furniture items and compromising layout completeness. To address these limitations, we propose RoomCraft, a multi-stage pipeline that converts real images, sketches, or text descriptions into coherent 3D indoor scenes. Our approach combines a scene generation pipeline with a constraint-driven optimization framework. The pipeline first extracts high-level scene information from user inputs and organizes it into a structured format containing room type, furniture items, and spatial relations. It then constructs a spatial relationship network to represent furniture arrangements and generates an optimized placement sequence using a heuristic-based depth-first search (HDFS) algorithm to ensure layout coherence. To handle complex multi-constraint scenarios, we introduce a unified constraint representation that processes both formal specifications and natural language inputs, enabling flexible constraint-oriented adjustments through a comprehensive action space design. Additionally, we propose a Conflict-Aware Positioning Strategy (CAPS) that dynamically adjusts placement weights to minimize furniture collisions and ensure layout completeness. Extensive experiments demonstrate that RoomCraft significantly outperforms existing methods in generating realistic, semantically coherent, and visually appealing room layouts across diverse input modalities.
Abstract (translated)
从用户输入生成逼真的3D室内场景在计算机视觉和图形学领域仍然是一个挑战性问题，需要精细地平衡几何一致性、空间关系以及视觉真实感。虽然神经网络生成方法常因全球空间推理能力有限而产生重复元素，程序化方法可以通过利用约束实现可控生成，但在面对多约束场景时却往往力不从心。当约束数量增多时，物体碰撞频繁发生，导致移除家具物品并损害布局完整性。为了克服这些限制，我们提出了RoomCraft，这是一种将真实图像、草图或文本描述转换为连贯3D室内场景的多阶段流水线方法。我们的方法结合了场景生成管道和基于约束驱动优化框架。 该管道首先从用户输入中提取高层次场景信息，并将其组织成包含房间类型、家具项目和空间关系的结构化格式。然后，它构建一个空间关系网络来表示家具布局安排，并使用启发式深度优先搜索（HDFS）算法生成最佳放置顺序以确保布局一致性。为了处理复杂的多约束场景，我们引入了一种统一的约束表示方法，可以同时处理正式规范和自然语言输入，并通过全面的动作设计实现灵活的基于约束调整。此外，我们还提出了一种冲突感知定位策略(CAPS)，该策略动态调整放置权重以尽量减少家具碰撞并确保布局完整性。 广泛的实验表明，RoomCraft在各种输入模式下显著优于现有方法，在生成逼真、语义连贯且视觉吸引人的房间布局方面表现出色。
URL
https://arxiv.org/abs/2506.22291
PDF
-
SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model
2025-06-27 07:35:04Shuhan Tan, John Lambert, Hong Jeon, Sakshum Kulshrestha, Yijing Bai, Jing Luo, Dragomir Anguelov, Mingxing Tan, Chiyu Max JiangAbstract
The goal of traffic simulation is to augment a potentially limited amount of manually-driven miles that is available for testing and validation, with a much larger amount of simulated synthetic miles. The culmination of this vision would be a generative simulated city, where given a map of the city and an autonomous vehicle (AV) software stack, the simulator can seamlessly simulate the trip from point A to point B by populating the city around the AV and controlling all aspects of the scene, from animating the dynamic agents (e.g., vehicles, pedestrians) to controlling the traffic light states. We refer to this vision as CitySim, which requires an agglomeration of simulation technologies: scene generation to populate the initial scene, agent behavior modeling to animate the scene, occlusion reasoning, dynamic scene generation to seamlessly spawn and remove agents, and environment simulation for factors such as traffic lights. While some key technologies have been separately studied in various works, others such as dynamic scene generation and environment simulation have received less attention in the research community. We propose SceneDiffuser++, the first end-to-end generative world model trained on a single loss function capable of point A-to-B simulation on a city scale integrating all the requirements above. We demonstrate the city-scale traffic simulation capability of SceneDiffuser++ and study its superior realism under long simulation conditions. We evaluate the simulation quality on an augmented version of the Waymo Open Motion Dataset (WOMD) with larger map regions to support trip-level simulation.
Abstract (translated)
交通模拟的目标是通过大量的仿真合成里程来增强有限的手动驾驶测试和验证里程。这一愿景的最终目标是一个生成式的虚拟城市，即给定一个城市的地图和自动驾驶车辆（AV）软件堆栈后，该模拟器能够无缝地从A点到B点进行旅程模拟，并且可以通过填充城市环境并控制场景的所有方面来实现这一点，包括动画化动态代理（例如，车辆、行人）以及管理交通灯状态等。我们将这一愿景称为CitySim，这需要一系列的仿真技术：用于初始化场景生成的技术，用于动画化场景的行为建模技术，遮挡推理，无缝启动和移除代理的动态场景生成技术，以及包括交通灯在内的环境模拟技术。尽管一些关键技术已经在各种研究中分别进行了单独的研究，但其他技术如动态场景生成和环境模拟在学术界却受到了较少的关注。 我们提出了SceneDiffuser++，这是第一个基于单个损失函数训练完成的端到端生成式世界模型，能够在城市规模上实现从A点到B点的仿真，并集成了上述所有需求。我们展示了SceneDiffuser++的城市级交通仿真的能力以及其在长时间模拟条件下的优越现实性。我们在Waymo开放运动数据集（WOMD）的一个增强版本上评估了该模拟的质量，此版本包含了更大的地图区域以支持行程级别的模拟。
URL
https://arxiv.org/abs/2506.21976
PDF
-
XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation
2025-06-26 16:04:16Bowen Chen, Mengyi Zhao, Haomiao Sun, Li Chen, Xu Wang, Kang Du, Xinglong WuAbstract
Achieving fine-grained control over subject identity and semantic attributes (pose, style, lighting) in text-to-image generation, particularly for multiple subjects, often undermines the editability and coherence of Diffusion Transformers (DiTs). Many approaches introduce artifacts or suffer from attribute entanglement. To overcome these challenges, we propose a novel multi-subject controlled generation model XVerse. By transforming reference images into offsets for token-specific text-stream modulation, XVerse allows for precise and independent control for specific subject without disrupting image latents or features. Consequently, XVerse offers high-fidelity, editable multi-subject image synthesis with robust control over individual subject characteristics and semantic attributes. This advancement significantly improves personalized and complex scene generation capabilities.
Abstract (translated)
在从文本生成图像的过程中，实现对主题身份和语义属性（如姿态、风格、光照）的精细控制，特别是在涉及多个主体时，通常会削弱扩散变压器（DiT）的可编辑性和连贯性。许多方法引入了伪影或遭受属性纠缠的问题。为了解决这些挑战，我们提出了一种新的多主题可控生成模型XVerse。通过将参考图像转换为用于特定标记文本流调节的偏移量，XVerse允许对特定主体进行精确且独立的控制，而不破坏图像潜在特征。因此，XVerse提供了高保真度、可编辑的多主体图像合成，并具有强大的个体主体特性和语义属性控制能力。这一进展显著提高了个性化和复杂场景生成的能力。
URL
https://arxiv.org/abs/2506.21416
PDF
-
WonderFree: Enhancing Novel View Quality and Cross-View Consistency for 3D Scene Exploration
2025-06-25 16:28:40Chaojun Ni, Jie Li, Haoyun Li, Hengyu Liu, Xiaofeng Wang, Zheng Zhu, Guosheng Zhao, Boyuan Wang, Chenxin Li, Guan Huang, Wenjun MeiAbstract
Interactive 3D scene generation from a single image has gained significant attention due to its potential to create immersive virtual worlds. However, a key challenge in current 3D generation methods is the limited explorability, which cannot render high-quality images during larger maneuvers beyond the original viewpoint, particularly when attempting to move forward into unseen areas. To address this challenge, we propose WonderFree, the first model that enables users to interactively generate 3D worlds with the freedom to explore from arbitrary angles and directions. Specifically, we decouple this challenge into two key subproblems: novel view quality, which addresses visual artifacts and floating issues in novel views, and cross-view consistency, which ensures spatial consistency across different viewpoints. To enhance rendering quality in novel views, we introduce WorldRestorer, a data-driven video restoration model designed to eliminate floaters and artifacts. In addition, a data collection pipeline is presented to automatically gather training data for WorldRestorer, ensuring it can handle scenes with varying styles needed for 3D scene generation. Furthermore, to improve cross-view consistency, we propose ConsistView, a multi-view joint restoration mechanism that simultaneously restores multiple perspectives while maintaining spatiotemporal coherence. Experimental results demonstrate that WonderFree not only enhances rendering quality across diverse viewpoints but also significantly improves global coherence and consistency. These improvements are confirmed by CLIP-based metrics and a user study showing a 77.20% preference for WonderFree over WonderWorld enabling a seamless and immersive 3D exploration experience. The code, model, and data will be publicly available.
Abstract (translated)
从单张图像生成交互式3D场景因其在创建沉浸式虚拟世界方面的潜力而备受关注。然而，目前的3D生成方法面临的一个关键挑战是探索性有限，在超出原始视点进行更大范围操作（尤其是尝试进入未见区域时）无法渲染高质量图片。为了解决这一难题，我们提出了WonderFree模型，这是首个允许用户以任意角度和方向互动地生成3D世界的模型。具体来说，我们将此挑战分解成两个关键子问题：新颖视角质量和跨视图一致性。前者关注于解决新颖视角中的视觉伪影和漂浮问题；后者则确保不同视角间的空间一致性。 为了提升新视角下的渲染质量，我们引入了WorldRestorer，这是一个基于数据驱动的视频修复模型，旨在消除漂浮物体及其它视觉伪影。此外，我们还提出了一种自动收集训练数据的数据采集管道，为WorldRestorer提供涵盖各种风格场景所需的数据，以适应3D场景生成的需求。 为了提高跨视图的一致性，我们提出了ConsistView，这是一种多视角联合修复机制，在同时恢复多个视角的同时保持时空一致性。实验结果表明，WonderFree不仅在不同视角下提升了渲染质量，而且还显著提高了全局一致性和连贯性。这些改进通过CLIP（一种基于对比学习的预训练模型）基准测试和用户研究得到了验证，结果显示有77.20%的用户更偏好使用WonderFree而非WonderWorld来获得无缝且沉浸式的3D探索体验。 该方法的相关代码、模型及数据将公开发布。
URL
https://arxiv.org/abs/2506.20590
PDF
