Scene_Generation
-
OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder
2026-03-17 03:43:37Sensen Gao, Zhaoqing Wang, Qihang Cao, Dongdong Yu, Changhu Wang, Tongliang Liu, Mingming Gong, Jiawang BianAbstract
Existing diffusion-based 3D scene generation methods primarily operate in 2D image/video latent spaces, which makes maintaining cross-view appearance and geometric consistency inherently challenging. To bridge this gap, we present OneWorld, a framework that performs diffusion directly within a coherent 3D representation space. Central to our approach is the 3D Unified Representation Autoencoder (3D-URAE); it leverages pretrained 3D foundation models and augments their geometry-centric nature by injecting appearance and distilling semantics into a unified 3D latent space. Furthermore, we introduce token-level Cross-View-Correspondence (CVC) consistency loss to explicitly enforce structural alignment across views, and propose Manifold-Drift Forcing (MDF) to mitigate train-inference exposure bias and shape a robust 3D manifold by mixing drifted and original representations. Comprehensive experiments demonstrate that OneWorld generates high-quality 3D scenes with superior cross-view consistency compared to state-of-the-art 2D-based methods. Our code will be available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16099
PDF
-
Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation
2026-03-16 07:59:39Xingtai Gui, Meijie Zhang, Tianyi Yan, Wencheng Han, Jiahao Gong, Feiyang Tan, Cheng-zhong Xu, Jianbing ShenAbstract
End-to-end autonomous driving aims to generate safe and plausible planning policies from raw sensor input. Driving world models have shown great potential in learning rich representations by predicting the future evolution of a driving scene. However, existing driving world models primarily focus on visual scene representation, and motion representation is not explicitly designed to be planner-shared and inheritable, leaving a schism between the optimization of visual scene generation and the requirements of precise motion planning. We present WorldDrive, a holistic framework that couples scene generation and real-time planning via unifying vision and motion representation. We first introduce a Trajectory-aware Driving World Model, which conditions on a trajectory vocabulary to enforce consistency between visual dynamics and motion intentions, enabling the generation of diverse and plausible future scenes conditioned on a specific trajectory. We transfer the vision and motion encoders to a downstream Multi-modal Planner, ensuring the driving policy operates on mature representations pre-optimized by scene generation. A simple interaction between motion representation, visual representation, and ego status can generate high-quality, multi-modal trajectories. Furthermore, to exploit the world model's foresight, we propose a Future-aware Rewarder, which distills future latent representation from the frozen world model to evaluate and select optimal trajectories in real-time. Extensive experiments on the NAVSIM, NAVSIM-v2, and nuScenes benchmarks demonstrate that WorldDrive achieves leading planning performance among vision-only methods while maintaining high-fidelity action-controlled video generation capabilities, providing strong evidence for the effectiveness of unifying vision and motion representation for robust autonomous driving.
Abstract (translated)
URL
https://arxiv.org/abs/2603.14948
PDF
-
Scene Generation at Absolute Scale: Utilizing Semantic and Geometric Guidance From Text for Accurate and Interpretable 3D Indoor Scene Generation
2026-03-14 11:46:27Stefan Ainetter, Thomas Deixelberger, Edoardo A. Dominici, Philipp Drescher, Konstantinos Vardis, Markus SteinbergerAbstract
We present GuidedSceneGen, a text-to-3D generation framework that produces metrically accurate, globally consistent, and semantically interpretable indoor scenes. Unlike prior text-driven methods that often suffer from geometric drift or scale ambiguity, our approach maintains an absolute world coordinate frame throughout the entire generation process. Starting from a textual scene description, we predict a global 3D layout encoding both semantic and geometric structure, which serves as a guiding proxy for downstream stages. A semantics- and depth-conditioned panoramic diffusion model then synthesizes 360° imagery aligned with the global layout, substantially improving spatial coherence. To explore unobserved regions, we employ a video diffusion model guided by optimized camera trajectories that balances coverage and collision avoidance, achieving up to 10x faster sampling compared to exhaustive path exploration. The generated views are fused using 3D Gaussian Splatting, yielding a consistent and fully navigable 3D scene in absolute scale. GuidedSceneGen enables accurate transfer of object poses and semantic labels from layout to reconstruction, and supports progressive scene expansion without re-alignment. Quantitative results and a user study demonstrate greater 3D consistency and layout plausibility compared to recent panoramic text-to-3D baselines.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13910
PDF
-
SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generation
2026-03-12 17:55:07Jun Luo, Jiaxiang Tang, Ruijie Lu, Gang ZengAbstract
Text-to-3D scene generation from natural language is highly desirable for digital content creation. However, existing methods are largely domain-restricted or reliant on predefined spatial relationships, limiting their capacity for unconstrained, open-vocabulary 3D scene synthesis. In this paper, we introduce SceneAssistant, a visual-feedback-driven agent designed for open-vocabulary 3D scene generation. Our framework leverages modern 3D object generation model along with the spatial reasoning and planning capabilities of Vision-Language Models (VLMs). To enable open-vocabulary scene composition, we provide the VLMs with a comprehensive set of atomic operations (e.g., Scale, Rotate, FocusOn). At each interaction step, the VLM receives rendered visual feedback and takes actions accordingly, iteratively refining the scene to achieve more coherent spatial arrangements and better alignment with the input text. Experimental results demonstrate that our method can generate diverse, open-vocabulary, and high-quality 3D scenes. Both qualitative analysis and quantitative human evaluations demonstrate the superiority of our approach over existing methods. Furthermore, our method allows users to instruct the agent to edit existing scenes based on natural language commands. Our code is available at this https URL
Abstract (translated)
从文本生成三维场景是数字内容创作中非常有价值的需求。然而，现有的方法大多局限于特定领域或者依赖于预定义的空间关系，这限制了它们在不受限、开放式词汇表3D场景合成中的应用能力。在这篇论文中，我们介绍了SceneAssistant，这是一个专为开放词汇表的3D场景生成设计的视觉反馈驱动代理。我们的框架结合了现代三维对象生成模型以及Vision-Language Models（VLMs）的空间推理和规划功能。为了支持开放式词汇表的场景组合，我们为VLM提供了全面的基础操作集（例如：缩放、旋转、聚焦等）。在每个交互步骤中，当接收到渲染后的视觉反馈时，VLM会相应地采取行动，并迭代优化场景以实现更连贯的空间排列和更好地与输入文本对齐。实验结果表明，我们的方法可以生成多样化的、开放式词汇表的以及高质量的3D场景。无论是定性分析还是定量的人类评估都证明了我们这种方法优于现有技术。此外，我们的方法允许用户根据自然语言命令指示代理编辑现有的场景。我们的代码可以在提供的网址上获取。 这种翻译旨在概述该论文的主要贡献和技术细节，并且提供了一个对非专业读者友好的版本。
URL
https://arxiv.org/abs/2603.12238
PDF
-
MANSION: Multi-floor lANguage-to-3D Scene generatIOn for loNg-horizon tasks
2026-03-12 05:22:42Lirong Che, Shuo Wen, Shan Huang, Chuang Wang, Yuzhe Yang, Gregory Dudek, Xueqian Wang, Jian SuAbstract
Real-world robotic tasks are long-horizon and often span multiple floors, demanding rich spatial reasoning. However, existing embodied benchmarks are largely confined to single-floor in-house environments, failing to reflect the complexity of real-world tasks. We introduce MANSION, the first language-driven framework for generating building-scale, multi-floor 3D environments. Being aware of vertical structural constraints, MANSION generates realistic, navigable whole-building structures with diverse, human-friendly scenes, enabling the development and evaluation of cross-floor long-horizon tasks. Building on this framework, we release MansionWorld, a dataset of over 1,000 diverse buildings ranging from hospitals to offices, alongside a Task-Semantic Scene Editing Agent that customizes these environments using open-vocabulary commands to meet specific user needs. Benchmarking reveals that state-of-the-art agents degrade sharply in our settings, establishing MANSION as a critical testbed for the next generation of spatial reasoning and planning.
Abstract (translated)
现实中的机器人任务通常涉及较长的时间跨度，并且往往横跨多层楼，需要复杂的空间推理能力。然而，现有的嵌入式基准测试大多局限于单层室内环境，无法反映实际任务的复杂性。为此，我们引入了MANSION，这是一个首屈一指的语言驱动框架，用于生成建筑规模、多楼层的3D环境。考虑到垂直结构约束，MANSION能够生成现实主义强且可导航的整体建筑结构，并包含多样化的人类友好型场景，从而支持跨楼层长时任务的发展和评估。 基于此框架，我们发布了MansionWorld数据集，其中包含了超过1000个多样化的建筑物模型，从医院到办公室应有尽有。此外，我们还提供了一个任务语义场景编辑代理，它能够通过开放式词汇命令自定义这些环境以满足特定用户需求。基准测试表明，在我们的设置中，最先进的代理性能明显下降，这证明MANSION已成为下一代空间推理和规划技术的重要测试平台。
URL
https://arxiv.org/abs/2603.11554
PDF
-
Pano3DComposer: Feed-Forward Compositional 3D Scene Generation from Single Panoramic Image
Abstract
Current compositional image-to-3D scene generation approaches construct 3D scenes by time-consuming iterative layout optimization or inflexible joint object-layout generation. Moreover, most methods rely on limited field-of-view perspective images, hindering the creation of complete 360-degree environments. To address these limitations, we design Pano3DComposer, an efficient feed-forward framework for panoramic images. To decouple object generation from layout estimation, we propose a plug-and-play Object-World Transformation Predictor. This module converts the 3D objects generated by off-the-shelf image-to-3D models from local to world coordinates. To achieve this, we adapt the VGGT architecture to Alignment-VGGT by using target object crop, multi-view object renderings and camera parameters to predict the transformation. The predictor is trained using pseudo-geometric supervision to address the shape discrepancy between generated and ground-truth objects. For input images from unseen domains, we further introduce a Coarse-to-Fine (C2F) alignment mechanism for Pano3DComposer that iteratively refines geometric consistency with feedback of scene rendering. Our method achieves superior geometric accuracy for image/text-to-3D tasks on synthetic and real-world datasets. It can generate a high-fidelity 3D scene in approximately 20 seconds on an RTX 4090 GPU. Project page: this https URL.
Abstract (translated)
当前的图像到三维场景生成方法通过耗时的迭代布局优化或刚性的联合对象-布局生成来构建3D场景。此外，大多数方法依赖于视野有限的透视图图像，这阻碍了创建完整的360度环境。为了解决这些限制，我们设计了一种面向全景图像的有效前馈框架Pano3DComposer。为了将物体生成与布局估计解耦，我们提出了一个即插即用的对象-世界变换预测器。该模块可以将现成的图像到三维模型产生的局部坐标中的三维对象转换为世界坐标系。为此，我们将VGGT架构通过使用目标物裁剪、多视角物体渲染以及相机参数调整为Alignment-VGGT来预测转换。我们利用伪几何监督训练预测器以解决生成和地面真实物体之间的形状差异问题。 对于来自未知领域的输入图像，我们进一步为Pano3DComposer引入了一种粗到细（C2F）对齐机制，该机制通过反馈场景渲染迭代地细化了几何一致性。我们的方法在合成数据集和现实世界数据集中实现了比现有技术更优越的几何精度，在将图片或文本转换成三维模型任务上表现尤为出色。它可以在NVIDIA RTX 4090 GPU上大约20秒内生成一个高保真的3D场景。 项目页面：[这里插入链接，原文中的链接无法直接在此显示]
URL
https://arxiv.org/abs/2603.05908
PDF
-
Narrative Weaver: Towards Controllable Long-Range Visual Consistency with Multi-Modal Conditioning
2026-03-04 08:22:22Zhengjian Yao, Yongzhi Li, Xinyuan Gao, Quan Chen, Peng Jiang, Yanye LuAbstract
We present "Narrative Weaver", a novel framework that addresses a fundamental challenge in generative AI: achieving multi-modal controllable, long-range, and consistent visual content generation. While existing models excel at generating high-fidelity short-form visual content, they struggle to maintain narrative coherence and visual consistency across extended sequences - a critical limitation for real-world applications such as filmmaking and e-commerce advertising. Narrative Weaver introduces the first holistic solution that seamlessly integrates three essential capabilities: fine-grained control, automatic narrative planning, and long-range coherence. Our architecture combines a Multimodal Large Language Model (MLLM) for high-level narrative planning with a novel fine-grained control module featuring a dynamic Memory Bank that prevents visual drift. To enable practical deployment, we develop a progressive, multi-stage training strategy that efficiently leverages existing pre-trained models, achieving state-of-the-art performance even with limited training data. Recognizing the absence of suitable evaluation benchmarks, we construct and release the E-commerce Advertising Video Storyboard Dataset (EAVSD) - the first comprehensive dataset for this task, containing over 330K high-quality images with rich narrative annotations. Through extensive experiments across three distinct scenarios (controllable multi-scene generation, autonomous storytelling, and e-commerce advertising), we demonstrate our method's superiority while opening new possibilities for AI-driven content creation.
Abstract (translated)
我们介绍了“叙事织工”(Narrative Weaver)，这是一个新颖的框架，旨在解决生成式人工智能中的一个基本挑战：实现多模态可控、长跨度和视觉内容的一致性生成。虽然现有的模型在生成高保真度的短格式视觉内容方面表现出色，但在扩展序列中保持叙述连贯性和视觉一致性方面存在困难——这对于电影制作和电子商务广告等现实世界应用而言是一个关键限制。“叙事织工”引入了第一个全面解决方案，无缝整合三种基本能力：细粒度控制、自动叙述规划以及长跨度的一致性。我们的架构结合了一种多模态大型语言模型（MLLM）用于高层次的叙述规划，并配备了一个具有动态记忆库的新颖细粒度控制模块，以防止视觉漂移。 为了实现实际部署，“叙事织工”开发了一种渐进式的、多阶段训练策略，高效利用现有的预训练模型，在数据有限的情况下也能达到最先进的性能。由于缺乏合适的评估基准，“我们构建并发布了电子商务广告视频故事版数据集（EAVSD）——这是该任务的第一个综合性数据集，包含超过33万张高质量图像及丰富的叙述注释。 通过在三种不同的场景中进行广泛的实验（可控多场景生成、自主叙事和电子商务广告），我们展示了“叙事织工”的优越性能，并开辟了人工智能驱动内容创作的新可能。
URL
https://arxiv.org/abs/2603.06688
PDF
-
WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memories
2026-03-02 16:36:56Yisu Zhang, Chenjie Cao, Tengfei Wang, Xuhui Zuo, Junta Wu, Jianke Zhu, Chunchao GuoAbstract
Recent advances in foundational Video Diffusion Models (VDMs) have yielded significant progress. Yet, despite the remarkable visual quality of generated videos, reconstructing consistent 3D scenes from these outputs remains challenging, due to limited camera controllability and inconsistent generated content when viewed from distinct camera trajectories. In this paper, we propose WorldStereo, a novel framework that bridges camera-guided video generation and 3D reconstruction via two dedicated geometric memory modules. Formally, the global-geometric memory enables precise camera control while injecting coarse structural priors through incrementally updated point clouds. Moreover, the spatial-stereo memory constrains the model's attention receptive fields with 3D correspondence to focus on fine-grained details from the memory bank. These components enable WorldStereo to generate multi-view-consistent videos under precise camera control, facilitating high-quality 3D reconstruction. Furthermore, the flexible control branch-based WorldStereo shows impressive efficiency, benefiting from the distribution matching distilled VDM backbone without joint training. Extensive experiments across both camera-guided video generation and 3D reconstruction benchmarks demonstrate the effectiveness of our approach. Notably, we show that WorldStereo acts as a powerful world model, tackling diverse scene generation tasks (whether starting from perspective or panoramic images) with high-fidelity 3D results. Models will be released.
Abstract (translated)
最近，基础视频扩散模型（VDMs）取得了显著进展。尽管这些模型生成的视频具有出色的视觉质量，但由于相机控制能力有限以及从不同摄像机轨迹观看时内容的一致性较差，重建连贯的三维场景仍然面临挑战。 本文提出了一个新框架WorldStereo，该框架通过两个专门的几何记忆模块将基于相机指导的视频生成与3D重建连接起来。具体而言，全局几何记忆模块实现了精确的相机控制，并且通过增量更新点云来注入粗略的结构先验知识；而空间立体记忆则通过使用三维对应关系约束模型的关注力接收域，使模型能够从内存库中关注细粒度的细节。 这些组件使得WorldStereo能够在精确控制摄像机的情况下生成多视角一致性的视频，并促进了高质量的3D重建。此外，基于灵活控制分支的WorldStereo表现出令人印象深刻的效率，在无需联合训练的前提下受益于分布匹配蒸馏VDM骨干网络。通过针对基于相机指导的视频生成和3D重建基准进行广泛的实验，展示了我们方法的有效性。 值得注意的是，结果显示WorldStereo充当了一个强大的世界模型，能够处理各种场景生成任务（无论从透视图还是全景图像开始），并以高保真度获得三维结果。相关模型将对外发布。
URL
https://arxiv.org/abs/2603.02049
PDF
-
DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving
2026-03-02 09:12:40Enhui Ma, Jiahuan Zhang, Guantian Zheng, Tao Tang, Shengbo Eben Li, Yuhang Lu, Xia Zhou, Xueyang Zhang, Yifei Zhan, Kun Zhan, Zhihui Hao, Xianpeng Lang, Kaicheng YuAbstract
Multimodal Large Language Models (MLLMs) are rapidly becoming the intelligence brain of end-to-end autonomous driving systems. A key challenge is to assess whether MLLMs can truly understand and follow complex real-world traffic rules. However, existing benchmarks mainly focus on single-rule scenarios like traffic sign recognition, neglecting the complexity of multi-rule concurrency and conflicts in real driving. Consequently, models perform well on simple tasks but often fail or violate rules in real world complex situations. To bridge this gap, we propose DriveCombo, a text and vision-based benchmark for compositional traffic rule reasoning. Inspired by human drivers' cognitive development, we propose a systematic Five-Level Cognitive Ladder that evaluates reasoning from single-rule understanding to multi-rule integration and conflict resolution, enabling quantitative assessment across cognitive stages. We further propose a Rule2Scene Agent that maps language-based traffic rules to dynamic driving scenes through rule crafting and scene generation, enabling scene-level traffic rule visual reasoning. Evaluations of 14 mainstream MLLMs reveal performance drops as task complexity grows, particularly during rule conflicts. After splitting the dataset and fine-tuning on the training set, we further observe substantial improvements in both traffic rule reasoning and downstream planning capabilities. These results highlight the effectiveness of DriveCombo in advancing compliant and intelligent autonomous driving systems.
Abstract (translated)
多模态大型语言模型（MLLMs）正在迅速成为端到端自动驾驶系统的智能核心。一个关键挑战在于评估这些模型是否能够真正理解并遵守复杂的现实世界交通规则。然而，现有的基准测试主要集中在单一规则场景上，如交通标志识别，忽视了实际驾驶中多重规则共存和冲突的复杂性。因此，虽然模型在简单任务上的表现良好，但在真实世界的复杂情况下往往会出现失误或违反规则的现象。 为了弥合这一差距，我们提出了DriveCombo，这是一个基于文本和视觉的基准测试平台，旨在评估组合型交通规则推理能力。受人类驾驶员认知发展过程启发，我们提出了一套五级认知阶梯系统评价从单一规则理解到多重规则整合与冲突解决的能力，并能够跨认知阶段进行量化评估。此外，我们还提出了Rule2Scene Agent这一概念，通过制定规则和生成场景将基于语言的交通规则映射为动态驾驶场景，从而实现基于场景级别的视觉推理。 对14种主流MLLMs的评测显示，在任务复杂度增加时性能显著下降，特别是在处理规则冲突的情况下。进一步地，在分割数据集并在训练集上进行微调后，我们观察到了交通规则推理和下游规划能力方面的重要改进。这些结果凸显了DriveCombo在推动合规且智能的自动驾驶系统发展方面的有效性。
URL
https://arxiv.org/abs/2603.01637
PDF
-
SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generation
2026-02-26 09:19:59Ling Wang, Hao-Xiang Guo, Xinzhou Wang, Fuchun Sun, Kai Sun, Pengkun Liu, Hang Xiao, Zhong Wang, Guangyuan Fu, Eric Li, Yang Liu, Yikai WangAbstract
We introduce SceneTransporter, an end-to-end framework for structured 3D scene generation from a single image. While existing methods generate part-level 3D objects, they often fail to organize these parts into distinct instances in open-world scenes. Through a debiased clustering probe, we reveal a critical insight: this failure stems from the lack of structural constraints within the model's internal assignment mechanism. Based on this finding, we reframe the task of structured 3D scene generation as a global correlation assignment problem. To solve this, SceneTransporter formulates and solves an entropic Optimal Transport (OT) objective within the denoising loop of the compositional DiT model. This formulation imposes two powerful structural constraints. First, the resulting transport plan gates cross-attention to enforce an exclusive, one-to-one routing of image patches to part-level 3D latents, preventing entanglement. Second, the competitive nature of the transport encourages the grouping of similar patches, a process that is further regularized by an edge-based cost, to form coherent objects and prevent fragmentation. Extensive experiments show that SceneTransporter outperforms existing methods on open-world scene generation, significantly improving instance-level coherence and geometric fidelity. Code and models will be publicly available at this https URL.
Abstract (translated)
我们介绍了一种名为SceneTransporter的端到端框架，用于从单张图像生成结构化的3D场景。尽管现有方法能够生成部分级别的3D对象，但它们往往无法在开放世界场景中将这些部分组织成不同的实例。通过偏差修正聚类探针，我们揭示了一个关键见解：这种失败的原因在于模型内部分配机制缺乏结构性约束。基于这一发现，我们将结构化3D场景的生成任务重新定义为一个全局相关性分配问题。为此，SceneTransporter在组成式DiT模型去噪循环中制定了并解决了熵最优传输（OT）目标。这种设定施加了两个强大的结构约束：首先，由此产生的运输计划通过控制跨注意力机制来确保图像补丁与部分级别3D潜在特征之间一对一的专有路由，从而防止纠缠；其次，竞争性的运输过程促使相似补丁分组，并且该过程进一步通过基于边的成本进行正则化，以形成连贯的对象并防止碎片化。广泛的实验表明，在开放世界场景生成方面，SceneTransporter超越了现有方法，显著提高了实例级别的连贯性和几何准确性。代码和模型将在[这个链接](https://this-url.com)公开提供。
URL
https://arxiv.org/abs/2602.22785
PDF
-
DrivePTS: A Progressive Learning Framework with Textual and Structural Enhancement for Driving Scene Generation
2026-02-26 02:42:14Zhechao Wang, Yiming Zeng, Lufan Ma, Zeqing Fu, Chen Bai, Ziyao Lin, Cheng LuAbstract
Synthesis of diverse driving scenes serves as a crucial data augmentation technique for validating the robustness and generalizability of autonomous driving systems. Current methods aggregate high-definition (HD) maps and 3D bounding boxes as geometric conditions in diffusion models for conditional scene generation. However, implicit inter-condition dependency causes generation failures when control conditions change independently. Additionally, these methods suffer from insufficient details in both semantic and structural aspects. Specifically, brief and view-invariant captions restrict semantic contexts, resulting in weak background modeling. Meanwhile, the standard denoising loss with uniform spatial weighting neglects foreground structural details, causing visual distortions and blurriness. To address these challenges, we propose DrivePTS, which incorporates three key innovations. Firstly, our framework adopts a progressive learning strategy to mitigate inter-dependency between geometric conditions, reinforced by an explicit mutual information constraint. Secondly, a Vision-Language Model is utilized to generate multi-view hierarchical descriptions across six semantic aspects, providing fine-grained textual guidance. Thirdly, a frequency-guided structure loss is introduced to strengthen the model's sensitivity to high-frequency elements, improving foreground structural fidelity. Extensive experiments demonstrate that our DrivePTS achieves state-of-the-art fidelity and controllability in generating diverse driving scenes. Notably, DrivePTS successfully generates rare scenes where prior methods fail, highlighting its strong generalization ability.
Abstract (translated)
合成多样化的驾驶场景是验证自动驾驶系统鲁棒性和泛化能力的关键数据增强技术。目前的方法在扩散模型中聚合高精度地图和3D边界框作为几何条件来生成条件场景。然而，隐含的跨条件依赖性导致当控制条件独立变化时产生生成失败的问题。此外，这些方法在语义和结构细节方面存在不足。具体来说，简短且视图不变的描述限制了语义上下文，从而弱化了背景建模效果。同时，标准的空间均匀加权去噪损失忽略了前景结构细节，导致视觉失真和模糊。 为了解决这些问题，我们提出了DrivePTS方法，该方法集成了三个关键创新点。首先，我们的框架采用了一种渐进式学习策略来缓解几何条件之间的相互依赖性，并通过显式的互信息约束加以强化。其次，利用视觉-语言模型生成六个语义方面的多视角层次化描述，提供精细的文本指导。第三，引入了基于频率引导的结构损失以增强模型对高频元素的敏感度，提高前景结构的一致性和真实性。 广泛的实验表明，我们的DrivePTS在生成多样化驾驶场景时达到了最先进的保真度和可控性。特别值得注意的是，DrivePTS成功地生成了一些此前方法无法处理的罕见场景，这突显了其强大的泛化能力。
URL
https://arxiv.org/abs/2602.22549
PDF
-
HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles
2026-02-24 20:03:47Yifan Wang, Francesco Pittaluga, Zaid Tasneem, Chenyu You, Manmohan Chandraker, Ziyu JiangAbstract
Controllable driving scene generation is critical for realistic and scalable autonomous driving simulation, yet existing approaches struggle to jointly achieve photorealism and precise control. We introduce HorizonForge, a unified framework that reconstructs scenes as editable Gaussian Splats and Meshes, enabling fine-grained 3D manipulation and language-driven vehicle insertion. Edits are rendered through a noise-aware video diffusion process that enforces spatial and temporal consistency, producing diverse scene variations in a single feed-forward pass without per-trajectory optimization. To standardize evaluation, we further propose HorizonSuite, a comprehensive benchmark spanning ego- and agent-level editing tasks such as trajectory modifications and object manipulation. Extensive experiments show that Gaussian-Mesh representation delivers substantially higher fidelity than alternative 3D representations, and that temporal priors from video diffusion are essential for coherent synthesis. Combining these findings, HorizonForge establishes a simple yet powerful paradigm for photorealistic, controllable driving simulation, achieving an 83.4% user-preference gain and a 25.19% FID improvement over the second best state-of-the-art method. Project page: this https URL .
Abstract (translated)
可控的驾驶场景生成对于现实主义和可扩展性的自主驾驶仿真至关重要，然而现有的方法难以同时实现照片级真实感与精确控制。我们引入了HorizonForge，这是一个统一框架，它将场景重建为可编辑的高斯斑点（Gaussian Splats）和网格（Meshes），从而能够进行细致入微的三维操作，并通过语言驱动的方式插入车辆。这些修改通过一个噪声感知视频扩散过程来渲染，在该过程中强制执行空间和时间一致性，无需每个轨迹优化即可在一次前馈传递中生成多样化的场景变化。 为了标准化评估，我们进一步提出了HorizonSuite，这是一个全面的基准测试平台，涵盖了自我主体级别（ego-level）和代理级别(agent-level)的任务编辑任务，例如轨迹修改和物体操作。广泛的实验表明，高斯网格表示法比其他三维表示方法具有显著更高的保真度，并且来自视频扩散的时间先验对于连贯合成是必不可少的。 结合这些发现，HorizonForge确立了一个简单但强大的范式，用于实现逼真的、可控制的驾驶仿真，在用户偏好和FID（Fréchet Inception Distance）指标上分别比现有最佳方法高出83.4%和25.19%。项目页面：[此链接](https://this-url.com)。 请注意，最后一个句子中的链接需要您替换为实际项目网页的URL。
URL
https://arxiv.org/abs/2602.21333
PDF
-
One2Scene: Geometric Consistent Explorable 3D Scene Generation from a Single Image
2026-02-23 12:15:54Pengfei Wang, Liyi Chen, Zhiyuan Ma, Yanjun Guo, Guowen Zhang, Lei ZhangAbstract
Generating explorable 3D scenes from a single image is a highly challenging problem in 3D vision. Existing methods struggle to support free exploration, often producing severe geometric distortions and noisy artifacts when the viewpoint moves far from the original perspective. We introduce \textbf{One2Scene}, an effective framework that decomposes this ill-posed problem into three tractable sub-tasks to enable immersive explorable scene generation. We first use a panorama generator to produce anchor views from a single input image as initialization. Then, we lift these 2D anchors into an explicit 3D geometric scaffold via a generalizable, feed-forward Gaussian Splatting network. Instead of treating the panorama as a single image for reconstruction, we project it into multiple sparse anchor views and reformulate the reconstruction task as multi-view stereo matching, which allows us to leverage robust geometric priors learned from large-scale multi-view datasets. A bidirectional feature fusion module is used to enforce cross-view consistency, yielding an efficient and geometrically reliable scaffold. Finally, the scaffold serves as a strong prior for a novel view generator to produce photorealistic and geometrically accurate views at arbitrary cameras. By explicitly conditioning on a 3D-consistent scaffold to perform reconstruction, One2Scene works stably under large camera motions, supporting immersive scene exploration. Extensive experiments show that One2Scene substantially outperforms state-of-the-art methods in panorama depth estimation, feed-forward 360° reconstruction, and explorable 3D scene generation. Code and models will be released.
Abstract (translated)
从单张图像生成可探索的三维场景是三维视觉中的一个高度挑战性的问题。现有方法在支持自由探索时往往难以克服，在视角远离原始视点时，会产生严重的几何畸变和噪声伪影。我们引入了**One2Scene**，这是一个有效的框架，它将这一病态问题分解为三个可处理的子任务，以实现沉浸式的可探索场景生成。 首先，使用全景生成器从单张输入图像中创建锚定视图作为初始化。然后，通过一个泛化的前向高斯散射网络将这些二维锚点提升到显式的三维几何支撑结构。不同于将全景视为单一图像进行重建，我们将它投影为多个稀疏的锚定视图，并重新定义了重建任务为多视角立体匹配问题，这使我们能够利用从大规模多视角数据集中学习到的强大几何先验知识。 双向特征融合模块被用来强制执行跨视图一致性，从而生成一个高效且几何上可靠的支撑结构。最后，该支撑结构作为先验条件用于新颖视角生成器中，以便在任意相机位置生成逼真且几何准确的视图。 通过显式地基于三维一致的支撑结构进行重建，One2Scene 在面对大范围相机运动时能稳定工作，支持沉浸式的场景探索。广泛的实验表明，One2Scene 在全景深度估计、前向360°重建和可探索三维场景生成方面显著优于现有的最先进方法。代码与模型将随后发布。
URL
https://arxiv.org/abs/2602.19766
PDF
-
SceneTok: A Compressed, Diffusable Token Space for 3D Scenes
2026-02-21 16:08:17Mohammad Asim, Christopher Wewer, Jan Eric LenssenAbstract
We present SceneTok, a novel tokenizer for encoding view sets of scenes into a compressed and diffusable set of unstructured tokens. Existing approaches for 3D scene representation and generation commonly use 3D data structures or view-aligned fields. In contrast, we introduce the first method that encodes scene information into a small set of permutation-invariant tokens that is disentangled from the spatial grid. The scene tokens are predicted by a multi-view tokenizer given many context views and rendered into novel views by employing a light-weight rectified flow decoder. We show that the compression is 1-3 orders of magnitude stronger than for other representations while still reaching state-of-the-art reconstruction quality. Further, our representation can be rendered from novel trajectories, including ones deviating from the input trajectory, and we show that the decoder gracefully handles uncertainty. Finally, the highly-compressed set of unstructured latent scene tokens enables simple and efficient scene generation in 5 seconds, achieving a much better quality-speed trade-off than previous paradigms.
Abstract (translated)
我们介绍了SceneTok，这是一种新颖的编码器，用于将场景视图集压缩并分解为一组无结构化的、可扩散的标记。现有的三维场景表示和生成方法通常使用3D数据结构或与视角对齐的字段。相比之下，我们首次提出了一种方法，可以将场景信息编码成一小组排列不变性标记，并且这些标记独立于空间网格。通过给定多个上下文视图，SceneTok由一个多视图编码器预测出场景标记，并利用轻量级校正流解码器将其渲染为新视角。 我们展示了这种压缩方法比其他表示方法强1-3个数量级的压缩能力，同时仍能达到业界领先的重建质量。此外，我们的表示可以从新的轨迹进行渲染，包括与输入轨迹偏离的情况，并且显示了解码器能够优雅地处理不确定性。最后，高度压缩的一组无结构潜在场景标记使得在5秒内简单高效地生成场景成为可能，实现了比以往范式更好的质量和速度权衡。
URL
https://arxiv.org/abs/2602.18882
PDF
-
RoboLayout: Differentiable 3D Scene Generation for Embodied Agents
2026-02-18 08:17:05Ali ShamsaddinlouAbstract
Recent advances in vision language models (VLMs) have shown strong potential for spatial reasoning and 3D scene layout generation from open-ended language instructions. However, generating layouts that are not only semantically coherent but also feasible for interaction by embodied agents remains challenging, particularly in physically constrained indoor environments. In this paper, RoboLayout is introduced as an extension of LayoutVLM that augments the original framework with agent-aware reasoning and improved optimization stability. RoboLayout integrates explicit reachability constraints into a differentiable layout optimization process, enabling the generation of layouts that are navigable and actionable by embodied agents. Importantly, the agent abstraction is not limited to a specific robot platform and can represent diverse entities with distinct physical capabilities, such as service robots, warehouse robots, humans of different age groups, or animals, allowing environment design to be tailored to the intended agent. In addition, a local refinement stage is proposed that selectively reoptimizes problematic object placements while keeping the remainder of the scene fixed, improving convergence efficiency without increasing global optimization iterations. Overall, RoboLayout preserves the strong semantic alignment and physical plausibility of LayoutVLM while enhancing applicability to agent-centric indoor scene generation, as demonstrated by experimental results across diverse scene configurations.
Abstract (translated)
最近，视觉语言模型（VLM）在从开放式语言指令生成空间推理和3D场景布局方面展示了强大的潜力。然而，创建既语义连贯又能被具身代理交互的可行布局仍然具有挑战性，尤其是在物理受限的室内环境中。在这篇论文中，介绍了一种名为RoboLayout的新方法，它是对LayoutVLM框架的一种扩展，通过加入代理感知推理和优化稳定性改进来增强原模型的功能。RoboLayout将显式的可达性约束整合进可微布局优化过程中，从而生成能够被具身代理导航并执行的场景布局。 重要的是，这种代理抽象不仅限于特定的机器人平台，还可以表示具有不同物理能力的各种实体，例如服务型机器人、仓库机器人、各年龄段的人类或动物等，从而使环境设计可以适应不同的目标代理。此外，还提出了一种局部细化阶段，在该阶段中选择性地重新优化有问题的对象位置，同时保持场景其余部分不变，从而在不增加全局优化迭代次数的情况下提高了收敛效率。 总体而言，通过跨多种不同配置的实验结果证明，RoboLayout不仅保留了LayoutVLM的强大语义对齐和物理合理性，而且还增强了其应用于代理中心视角的室内场景生成的能力。
URL
https://arxiv.org/abs/2603.05522
PDF
-
DCDM: Divide-and-Conquer Diffusion Models for Consistency-Preserving Video Generation
2026-02-14 07:02:36Haoyu Zhao, Yuang Zhang, Junqi Cheng, Jiaxi Gu, Zenghui Lu, Peng Shu, Zuxuan Wu, Yu-Gang JiangAbstract
Recent video generative models have demonstrated impressive visual fidelity, yet they often struggle with semantic, geometric, and identity consistency. In this paper, we propose a system-level framework, termed the Divide-and-Conquer Diffusion Model (DCDM), to address three key challenges: (1) intra-clip world knowledge consistency, (2) inter-clip camera consistency, and (3) inter-shot element consistency. DCDM decomposes video consistency modeling under these scenarios into three dedicated components while sharing a unified video generation backbone. For intra-clip consistency, DCDM leverages a large language model to parse input prompts into structured semantic representations, which are subsequently translated into coherent video content by a diffusion transformer. For inter-clip camera consistency, we propose a temporal camera representation in the noise space that enables precise and stable camera motion control, along with a text-to-image initialization mechanism to further enhance controllability. For inter-shot consistency, DCDM adopts a holistic scene generation paradigm with windowed cross-attention and sparse inter-shot self-attention, ensuring long-range narrative coherence while maintaining computational efficiency. We validate our framework on the test set of the CVM Competition at AAAI'26, and the results demonstrate that the proposed strategies effectively address these challenges.
Abstract (translated)
URL
https://arxiv.org/abs/2602.13637
PDF
-
Code2Worlds: Empowering Coding LLMs for 4D World Generation
2026-02-12 09:34:28Yi Zhang, Yunshuang Wang, Zeyu Zhang, Hao TangAbstract
Achieving spatial intelligence requires moving beyond visual plausibility to build world simulators grounded in physical laws. While coding LLMs have advanced static 3D scene generation, extending this paradigm to 4D dynamics remains a critical frontier. This task presents two fundamental challenges: multi-scale context entanglement, where monolithic generation fails to balance local object structures with global environmental layouts; and a semantic-physical execution gap, where open-loop code generation leads to physical hallucinations lacking dynamic fidelity. We introduce Code2Worlds, a framework that formulates 4D generation as language-to-simulation code generation. First, we propose a dual-stream architecture that disentangles retrieval-augmented object generation from hierarchical environmental orchestration. Second, to ensure dynamic fidelity, we establish a physics-aware closed-loop mechanism in which a PostProcess Agent scripts dynamics, coupled with a VLM-Motion Critic that performs self-reflection to iteratively refine simulation code. Evaluations on the Code4D benchmark show Code2Worlds outperforms baselines with a 41% SGS gain and 49% higher Richness, while uniquely generating physics-aware dynamics absent in prior static methods. Code: this https URL. Website: this https URL.
Abstract (translated)
实现空间智能需要超越视觉逼真度，构建基于物理定律的现实世界模拟器。虽然编码大型语言模型（LLM）已经推进了静态3D场景生成的发展，但将这一范式扩展到4D动态领域仍然是一个关键的研究前沿。这项任务面临两个基本挑战：多尺度上下文纠缠问题，在这种情况下，单一的整体生成方法无法平衡局部物体结构和全局环境布局；以及语义-物理执行差距，开放循环代码生成会导致缺乏动态真实的物理幻觉。我们引入了Code2Worlds框架，该框架将4D生成作为语言到模拟代码生成来处理。 首先，我们提出了一种双流架构，它分离出增强检索的物体生成与分层环境协调。其次，为了确保动态真实度，我们建立了一个具有物理学感知能力的闭环机制，在这个机制中，后处理代理（PostProcess Agent）编写动力学脚本，并结合视觉语言模型-运动批评家（VLM-Motion Critic），该批评家执行自我反思以迭代地改进模拟代码。 在Code4D基准测试上的评估表明，Code2Worlds相较于基线方法，在SGS指标上提高了41%，而在丰富度方面提高了49%。同时，它还能独特生成以前静态方法中缺失的具有物理学感知能力的动力学效果。 请注意，原文末尾提及了代码链接和网站链接，但这些链接在您的问题中是以占位符的形式出现的（"this https URL."），因此我无法直接提供它们的具体内容或访问。如果您需要访问这些资源，请前往原发布页面获取详细信息。
URL
https://arxiv.org/abs/2602.11757
PDF
-
LLM-Driven 3D Scene Generation of Agricultural Simulation Environments
2026-02-12 08:33:01Arafa Yoncalik, Wouter Jansen, Nico Huebel, Mohammad Hasan Rahmani, Jan SteckelAbstract
Procedural generation techniques in 3D rendering engines have revolutionized the creation of complex environments, reducing reliance on manual design. Recent approaches using Large Language Models (LLMs) for 3D scene generation show promise but often lack domain-specific reasoning, verification mechanisms, and modular design. These limitations lead to reduced control and poor scalability. This paper investigates the use of LLMs to generate agricultural synthetic simulation environments from natural language prompts, specifically to address the limitations of lacking domain-specific reasoning, verification mechanisms, and modular design. A modular multi-LLM pipeline was developed, integrating 3D asset retrieval, domain knowledge injection, and code generation for the Unreal rendering engine using its API. This results in a 3D environment with realistic planting layouts and environmental context, all based on the input prompt and the domain knowledge. To enhance accuracy and scalability, the system employs a hybrid strategy combining LLM optimization techniques such as few-shot prompting, Retrieval-Augmented Generation (RAG), finetuning, and validation. Unlike monolithic models, the modular architecture enables structured data handling, intermediate verification, and flexible expansion. The system was evaluated using structured prompts and semantic accuracy metrics. A user study assessed realism and familiarity against real-world images, while an expert comparison demonstrated significant time savings over manual scene design. The results confirm the effectiveness of multi-LLM pipelines in automating domain-specific 3D scene generation with improved reliability and precision. Future work will explore expanding the asset hierarchy, incorporating real-time generation, and adapting the pipeline to other simulation domains beyond agriculture.
Abstract (translated)
三维渲染引擎中的程序生成技术已经彻底改变了复杂环境的创建，减少了对手动设计的依赖。最近使用大型语言模型（LLMs）进行3D场景生成的方法显示出潜力，但往往缺乏领域特定推理、验证机制和模块化设计能力。这些限制导致控制减弱和可扩展性差。本文探讨了利用LLM根据自然语言提示生成农业合成仿真环境的方法，以解决缺乏领域特定推理、验证机制和模块化设计的局限性。 为此，开发了一种模块化的多LLM管道，结合3D资产检索、领域知识注入以及使用Unreal渲染引擎API进行代码生成。这产生了一个基于输入提示和领域知识的真实种植布局和环境上下文的三维环境。 为了提高准确性和可扩展性，该系统采用了一种混合策略，结合了包括少量样本提示（few-shot prompting）、增强式检索生成（Retrieval-Augmented Generation, RAG）、微调和验证在内的LLM优化技术。与单一模型相比，模块化架构允许结构化数据处理、中间验证和灵活扩展。 通过使用结构化提示和语义准确性度量对系统进行了评估。一项用户研究评估了合成环境的真实感和熟悉程度，并将其与真实世界的图像进行比较；专家对比显示该方法比手动场景设计节省大量时间。结果证实了多LLM管道在自动化特定领域的3D场景生成方面的有效性，提高了可靠性和精度。 未来的工作将探索扩展资产层次结构、结合实时生成以及适应农业以外的其他模拟领域。
URL
https://arxiv.org/abs/2602.11706
PDF
-
SAGE: Scalable Agentic 3D Scene Generation for Embodied AI
2026-02-10 18:59:55Hongchi Xia, Xuan Li, Zhaoshuo Li, Qianli Ma, Jiashu Xu, Ming-Yu Liu, Yin Cui, Tsung-Yi Lin, Wei-Chiu Ma, Shenlong Wang, Shuran Song, Fangyin WeiAbstract
Real-world data collection for embodied agents remains costly and unsafe, calling for scalable, realistic, and simulator-ready 3D environments. However, existing scene-generation systems often rely on rule-based or task-specific pipelines, yielding artifacts and physically invalid scenes. We present SAGE, an agentic framework that, given a user-specified embodied task (e.g., "pick up a bowl and place it on the table"), understands the intent and automatically generates simulation-ready environments at scale. The agent couples multiple generators for layout and object composition with critics that evaluate semantic plausibility, visual realism, and physical stability. Through iterative reasoning and adaptive tool selection, it self-refines the scenes until meeting user intent and physical validity. The resulting environments are realistic, diverse, and directly deployable in modern simulators for policy training. Policies trained purely on this data exhibit clear scaling trends and generalize to unseen objects and layouts, demonstrating the promise of simulation-driven scaling for embodied AI. Code, demos, and the SAGE-10k dataset can be found on the project page here: this https URL.
Abstract (translated)
真实世界中为具身智能体（如机器人或虚拟角色）收集数据既昂贵又不安全，因此需要可扩展、逼真且适用于模拟器的3D环境。然而，现有的场景生成系统通常依赖于基于规则的方法或特定任务的管道，这会导致生成的效果不佳和物理上无效的场景。我们介绍了SAGE框架，这是一种智能体驱动的方法，能够根据用户指定的任务（例如，“拿起碗并把它放在桌子上”），理解意图，并自动生成大规模的、可用于模拟器的环境。 该方法集成了多个布局与物体组合的生成器以及评估语义合理性、视觉真实性和物理稳定性的批评者。通过迭代推理和适应性工具选择，智能体能够不断自我优化场景，直至符合用户意图并满足物理有效性要求。最终产生的环境既逼真又多样化，并且可以直接部署在现代模拟器中用于策略训练。 仅使用这种数据进行的策略训练显示出明确的规模效应，并能推广到未见过的对象和布局上，这表明了基于模拟的数据驱动扩展对具身AI的前景。该项目的相关代码、演示以及SAGE-10k数据集可以在项目页面（此链接）找到。
URL
https://arxiv.org/abs/2602.10116
PDF
-
MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation
2026-02-10 15:19:17Jiaxu Wang, Yicheng Jiang, Tianlun He, Jingkai Sun, Qiang Zhang, Junhao He, Jiahang Cao, Zesen Gan, Mingyuan Sun, Qiming Shao, Xiangyu YueAbstract
World-model-based imagine-then-act becomes a promising paradigm for robotic manipulation, yet existing approaches typically support either purely image-based forecasting or reasoning over partial 3D geometry, limiting their ability to predict complete 4D scene dynamics. This work proposes a novel embodied 4D world model that enables geometrically consistent, arbitrary-view RGBD generation: given only a single-view RGBD observation as input, the model imagines the remaining viewpoints, which can then be back-projected and fused to assemble a more complete 3D structure across time. To efficiently learn the multi-view, cross-modality generation, we explicitly design cross-view and cross-modality feature fusion that jointly encourage consistency between RGB and depth and enforce geometric alignment across views. Beyond prediction, converting generated futures into actions is often handled by inverse dynamics, which is ill-posed because multiple actions can explain the same transition. We address this with a test-time action optimization strategy that backpropagates through the generative model to infer a trajectory-level latent best matching the predicted future, and a residual inverse dynamics model that turns this trajectory prior into accurate executable actions. Experiments on three datasets demonstrate strong performance on both 4D scene generation and downstream manipulation, and ablations provide practical insights into the key design choices.
Abstract (translated)
基于世界模型的想象然后行动成为机器人操作的一个有前景的新范式，但现有方法通常仅支持纯图像预测或在部分3D几何结构上的推理，这限制了它们预测完整4D场景动态的能力。这项工作提出了一种新型的具身化4D世界模型，该模型能够生成任意视角下的一致RGBD数据：给定单一视角的RGBD观测输入后，模型可以想象其他视点，并将这些视图反投影融合以构建更完整的3D结构（跨时间）。为了高效地学习多视角、跨模态的数据生成，我们设计了跨视图和跨模态特征融合，从而共同促进RGB与深度数据之间的一致性和不同视图间的几何对齐。除了预测之外，通常通过逆动力学将生成的未来转换为动作执行，但由于多种动作可能解释相同的过渡过程，这种做法是不适定的问题。为此，我们提出了一种测试时的动作优化策略，该策略通过生成模型反向传播以推断出与预测未来最匹配的轨迹级潜在变量，并结合残差逆动力学模型将此轨迹先验转化为准确可执行的动作。 实验在三个数据集上进行，展示了在4D场景生成和下游操作任务上的强大性能。消融研究提供了对关键设计选择的实际见解。
URL
https://arxiv.org/abs/2602.09878
PDF
