Scene_Generation
-
Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models
2025-11-01 11:16:25Panwang Pan, Chenguo Lin, Jingjing Zhao, Chenxin Li, Yuchen Lin, Haopeng Li, Honglei Yan, Kairun Wen, Yunlong Lin, Yixuan Yuan, Yadong MuAbstract
We introduce Diff4Splat, a feed-forward method that synthesizes controllable and explicit 4D scenes from a single image. Our approach unifies the generative priors of video diffusion models with geometry and motion constraints learned from large-scale 4D datasets. Given a single input image, a camera trajectory, and an optional text prompt, Diff4Splat directly predicts a deformable 3D Gaussian field that encodes appearance, geometry, and motion, all in a single forward pass, without test-time optimization or post-hoc refinement. At the core of our framework lies a video latent transformer, which augments video diffusion models to jointly capture spatio-temporal dependencies and predict time-varying 3D Gaussian primitives. Training is guided by objectives on appearance fidelity, geometric accuracy, and motion consistency, enabling Diff4Splat to synthesize high-quality 4D scenes in 30 seconds. We demonstrate the effectiveness of Diff4Splatacross video generation, novel view synthesis, and geometry extraction, where it matches or surpasses optimization-based methods for dynamic scene synthesis while being significantly more efficient.
Abstract (translated)
URL
https://arxiv.org/abs/2511.00503
PDF
-
HiGS: Hierarchical Generative Scene Framework for Multi-Step Associative Semantic Spatial Composition
2025-10-31 03:50:47Jiacheng Hong, Kunzhen Wu, Mingrui Yu, Yichao Gu, Shengze Xue, Shuangjiu Xiao, Deli DongAbstract
Three-dimensional scene generation holds significant potential in gaming, film, and virtual reality. However, most existing methods adopt a single-step generation process, making it difficult to balance scene complexity with minimal user input. Inspired by the human cognitive process in scene modeling, which progresses from global to local, focuses on key elements, and completes the scene through semantic association, we propose HiGS, a hierarchical generative framework for multi-step associative semantic spatial composition. HiGS enables users to iteratively expand scenes by selecting key semantic objects, offering fine-grained control over regions of interest while the model completes peripheral areas automatically. To support structured and coherent generation, we introduce the Progressive Hierarchical Spatial-Semantic Graph (PHiSSG), which dynamically organizes spatial relationships and semantic dependencies across the evolving scene structure. PHiSSG ensures spatial and geometric consistency throughout the generation process by maintaining a one-to-one mapping between graph nodes and generated objects and supporting recursive layout optimization. Experiments demonstrate that HiGS outperforms single-stage methods in layout plausibility, style consistency, and user preference, offering a controllable and extensible paradigm for efficient 3D scene construction.
Abstract (translated)
URL
https://arxiv.org/abs/2510.27148
PDF
-
OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes
2025-10-30 17:59:51Yukun Huang, Jiwen Yu, Yanning Zhou, Jianan Wang, Xintao Wang, Pengfei Wan, Xihui LiuAbstract
There are two prevalent ways to constructing 3D scenes: procedural generation and 2D lifting. Among them, panorama-based 2D lifting has emerged as a promising technique, leveraging powerful 2D generative priors to produce immersive, realistic, and diverse 3D environments. In this work, we advance this technique to generate graphics-ready 3D scenes suitable for physically based rendering (PBR), relighting, and simulation. Our key insight is to repurpose 2D generative models for panoramic perception of geometry, textures, and PBR materials. Unlike existing 2D lifting approaches that emphasize appearance generation and ignore the perception of intrinsic properties, we present OmniX, a versatile and unified framework. Based on a lightweight and efficient cross-modal adapter structure, OmniX reuses 2D generative priors for a broad range of panoramic vision tasks, including panoramic perception, generation, and completion. Furthermore, we construct a large-scale synthetic panorama dataset containing high-quality multimodal panoramas from diverse indoor and outdoor scenes. Extensive experiments demonstrate the effectiveness of our model in panoramic visual perception and graphics-ready 3D scene generation, opening new possibilities for immersive and physically realistic virtual world generation.
Abstract (translated)
URL
https://arxiv.org/abs/2510.26800
PDF
-
Token Is All You Need: Cognitive Planning through Sparse Intent Alignment
2025-10-30 12:16:45Shiyao SangAbstract
We challenge the long-standing assumption that exhaustive scene modeling is required for high-performance end-to-end autonomous driving (E2EAD). Unlike world-model approaches that rely on computationally intensive future scene generation or vision-language-action (VLA) systems constrained by Markov assumptions, we show that a minimal set of semantically rich tokens is sufficient for effective planning. Experiments on the nuPlan benchmark (720 scenarios, over 11,000 samples) using perception-informed BEV representations yield three key findings: (1) even without future prediction, our sparse representation achieves 0.548 m ADE, comparable to or surpassing prior methods reporting around 0.75 m on nuScenes; (2) conditioning trajectory decoding on predicted future tokens reduces ADE to 0.479 m, a 12.6% improvement over current-state baselines; and (3) explicit reconstruction loss offers no benefit and may degrade performance under reliable perception inputs. Notably, we observe the emergence of temporal fuzziness, where the model adaptively attends to task-relevant semantics rather than aligning rigidly to fixed timestamps, providing a cognitive advantage for planning under uncertainty. Our "token is all you need" principle marks a paradigm shift from reconstructing the world to understanding it, laying a foundation for cognitively inspired systems that plan through imagination rather than reaction.
Abstract (translated)
URL
https://arxiv.org/abs/2511.05540
PDF
-
TRELLISWorld: Training-Free World Generation from Object Generators
2025-10-27 21:40:31Hanke Chen, Yuan Liu, Minchen LiAbstract
Text-driven 3D scene generation holds promise for a wide range of applications, from virtual prototyping to AR/VR and simulation. However, existing methods are often constrained to single-object generation, require domain-specific training, or lack support for full 360-degree viewability. In this work, we present a training-free approach to 3D scene synthesis by repurposing general-purpose text-to-3D object diffusion models as modular tile generators. We reformulate scene generation as a multi-tile denoising problem, where overlapping 3D regions are independently generated and seamlessly blended via weighted averaging. This enables scalable synthesis of large, coherent scenes while preserving local semantic control. Our method eliminates the need for scene-level datasets or retraining, relies on minimal heuristics, and inherits the generalization capabilities of object-level priors. We demonstrate that our approach supports diverse scene layouts, efficient generation, and flexible editing, establishing a simple yet powerful foundation for general-purpose, language-driven 3D scene construction.
Abstract (translated)
URL
https://arxiv.org/abs/2510.23880
PDF
-
Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method
2025-10-27 03:52:45Bohan Li, Xin Jin, Hu Zhu, Hongsi Liu, Ruikai Li, Jiazhe Guo, Kaiwen Cai, Chao Ma, Yueming Jin, Hao Zhao, Xiaokang Yang, Wenjun ZengAbstract
Driving scene generation is a critical domain for autonomous driving, enabling downstream applications, including perception and planning evaluation. Occupancy-centric methods have recently achieved state-of-the-art results by offering consistent conditioning across frames and modalities; however, their performance heavily depends on annotated occupancy data, which still remains scarce. To overcome this limitation, we curate Nuplan-Occ, the largest semantic occupancy dataset to date, constructed from the widely used Nuplan benchmark. Its scale and diversity facilitate not only large-scale generative modeling but also autonomous driving downstream applications. Based on this dataset, we develop a unified framework that jointly synthesizes high-quality semantic occupancy, multi-view videos, and LiDAR point clouds. Our approach incorporates a spatio-temporal disentangled architecture to support high-fidelity spatial expansion and temporal forecasting of 4D dynamic occupancy. To bridge modal gaps, we further propose two novel techniques: a Gaussian splatting-based sparse point map rendering strategy that enhances multi-view video generation, and a sensor-aware embedding strategy that explicitly models LiDAR sensor properties for realistic multi-LiDAR simulation. Extensive experiments demonstrate that our method achieves superior generation fidelity and scalability compared to existing approaches, and validates its practical value in downstream tasks. Repo: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2510.22973
PDF
-
WorldGrow: Generating Infinite 3D World
2025-10-24 17:39:52Sikuang Li, Chen Yang, Jiemin Fang, Taoran Yi, Jia Lu, Jiazhong Cen, Lingxi Xie, Wei Shen, Qi TianAbstract
We tackle the challenge of generating the infinitely extendable 3D world -- large, continuous environments with coherent geometry and realistic appearance. Existing methods face key challenges: 2D-lifting approaches suffer from geometric and appearance inconsistencies across views, 3D implicit representations are hard to scale up, and current 3D foundation models are mostly object-centric, limiting their applicability to scene-level generation. Our key insight is leveraging strong generation priors from pre-trained 3D models for structured scene block generation. To this end, we propose WorldGrow, a hierarchical framework for unbounded 3D scene synthesis. Our method features three core components: (1) a data curation pipeline that extracts high-quality scene blocks for training, making the 3D structured latent representations suitable for scene generation; (2) a 3D block inpainting mechanism that enables context-aware scene extension; and (3) a coarse-to-fine generation strategy that ensures both global layout plausibility and local geometric/textural fidelity. Evaluated on the large-scale 3D-FRONT dataset, WorldGrow achieves SOTA performance in geometry reconstruction, while uniquely supporting infinite scene generation with photorealistic and structurally consistent outputs. These results highlight its capability for constructing large-scale virtual environments and potential for building future world models.
Abstract (translated)
URL
https://arxiv.org/abs/2510.21682
PDF
-
AutoScape: Geometry-Consistent Long-Horizon Scene Generation
2025-10-23 16:44:34Jiacheng Chen, Ziyu Jiang, Mingfu Liang, Bingbing Zhuang, Jong-Chyi Su, Sparsh Garg, Ying Wu, Manmohan ChandrakerAbstract
This paper proposes AutoScape, a long-horizon driving scene generation framework. At its core is a novel RGB-D diffusion model that iteratively generates sparse, geometrically consistent keyframes, serving as reliable anchors for the scene's appearance and geometry. To maintain long-range geometric consistency, the model 1) jointly handles image and depth in a shared latent space, 2) explicitly conditions on the existing scene geometry (i.e., rendered point clouds) from previously generated keyframes, and 3) steers the sampling process with a warp-consistent guidance. Given high-quality RGB-D keyframes, a video diffusion model then interpolates between them to produce dense and coherent video frames. AutoScape generates realistic and geometrically consistent driving videos of over 20 seconds, improving the long-horizon FID and FVD scores over the prior state-of-the-art by 48.6\% and 43.0\%, respectively.
Abstract (translated)
本文提出了AutoScape，这是一种长时域驾驶场景生成框架。其核心是一个新颖的RGB-D扩散模型，该模型通过迭代生成稀疏、几何一致的关键帧，这些关键帧作为场景外观和几何结构的可靠锚点。为了维持长距离几何一致性，模型采用了以下策略：1）在共享潜在空间中同时处理图像和深度信息；2）明确地以先前生成的关键帧中存在的场景几何（即渲染后的点云）为条件；3）使用一致变形引导来指导采样过程。基于高质量的RGB-D关键帧，随后的视频扩散模型会在这之间进行插值运算，从而产生密集且连贯的视频帧。AutoScape能够生成超过20秒的真实感和几何一致性良好的驾驶视频，在长时域FID（Fréchet Inception Distance）和FVD（Feature Voxel Distance）评分上分别比之前的最佳方法提高了48.6% 和 43.0%。
URL
https://arxiv.org/abs/2510.20726
PDF
-
Seed3D 1.0: From Images to High-Fidelity Simulation-Ready 3D Assets
2025-10-22 18:16:32Jiashi Feng, Xiu Li, Jing Lin, Jiahang Liu, Gaohong Liu, Weiqiang Lou, Su Ma, Guang Shi, Qinlong Wang, Jun Wang, Zhongcong Xu, Xuanyu Yi, Zihao Yu, Jianfeng Zhang, Yifan Zhu, Rui Chen, Jinxin Chi, Zixian Du, Li Han, Lixin Huang, Kaihua Jiang, Yuhan Li, Guan Luo, Shuguang Wang, Qianyi Wu, Fan Yang, Junyang Zhang, Xuanmeng ZhangAbstract
Developing embodied AI agents requires scalable training environments that balance content diversity with physics accuracy. World simulators provide such environments but face distinct limitations: video-based methods generate diverse content but lack real-time physics feedback for interactive learning, while physics-based engines provide accurate dynamics but face scalability limitations from costly manual asset creation. We present Seed3D 1.0, a foundation model that generates simulation-ready 3D assets from single images, addressing the scalability challenge while maintaining physics rigor. Unlike existing 3D generation models, our system produces assets with accurate geometry, well-aligned textures, and realistic physically-based materials. These assets can be directly integrated into physics engines with minimal configuration, enabling deployment in robotic manipulation and simulation training. Beyond individual objects, the system scales to complete scene generation through assembling objects into coherent environments. By enabling scalable simulation-ready content creation, Seed3D 1.0 provides a foundation for advancing physics-based world simulators. Seed3D 1.0 is now available on this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2510.19944
PDF
-
DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusion
2025-10-17 03:00:08Weijie Wang, Jiagang Zhu, Zeyu Zhang, Xiaofeng Wang, Zheng Zhu, Guosheng Zhao, Chaojun Ni, Haoxiao Wang, Guan Huang, Xinze Chen, Yukun Zhou, Wenkang Qin, Duochao Shi, Haoyun Li, Guanghong Jia, Jiwen LuAbstract
We present DriveGen3D, a novel framework for generating high-quality and highly controllable dynamic 3D driving scenes that addresses critical limitations in existing methodologies. Current approaches to driving scene synthesis either suffer from prohibitive computational demands for extended temporal generation, focus exclusively on prolonged video synthesis without 3D representation, or restrict themselves to static single-scene reconstruction. Our work bridges this methodological gap by integrating accelerated long-term video generation with large-scale dynamic scene reconstruction through multimodal conditional control. DriveGen3D introduces a unified pipeline consisting of two specialized components: FastDrive-DiT, an efficient video diffusion transformer for high-resolution, temporally coherent video synthesis under text and Bird's-Eye-View (BEV) layout guidance; and FastRecon3D, a feed-forward reconstruction module that rapidly builds 3D Gaussian representations across time, ensuring spatial-temporal consistency. Together, these components enable real-time generation of extended driving videos (up to $424\times800$ at 12 FPS) and corresponding dynamic 3D scenes, achieving SSIM of 0.811 and PSNR of 22.84 on novel view synthesis, all while maintaining parameter efficiency.
Abstract (translated)
我们介绍了DriveGen3D，这是一个用于生成高质量、高度可控的动态3D驾驶场景的新颖框架，解决了现有方法中的关键限制。当前的驾驶场景合成方法要么在长时间生成时面临计算需求过高的问题，要么仅专注于没有3D表示的长时间视频合成都市仿真中常见的情况，或者局限于静态单一场景重建。 我们的工作通过结合加速长期视频生成和大规模动态场景重构来弥合这些方法论之间的差距，并且支持多模态条件控制。DriveGen3D引入了一个统一的工作流程，包括两个专门化的组件：FastDrive-DiT是一个高效的视频扩散变压器，在文本和鸟瞰图（BEV）布局引导下进行高分辨率、时间连贯的视频合成；以及FastRecon3D，这是一个前馈重构模块，可以快速构建时序中的三维高斯表示，并保证空间-时间一致性。这两个组件共同实现实时生成扩展驾驶视频（高达424x800像素，12 FPS）和相应的动态3D场景，在新视角合成中达到SSIM为0.811、PSNR为22.84的性能指标，同时保持参数效率。
URL
https://arxiv.org/abs/2510.15264
PDF
-
CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation
2025-10-16 03:29:06Li Liang, Bo Miao, Xinyu Wang, Naveed Akhtar, Jordan Vice, Ajmal MianAbstract
Outdoor 3D semantic scene generation produces realistic and semantically rich environments for applications such as urban simulation and autonomous driving. However, advances in this direction are constrained by the absence of publicly available, well-annotated datasets. We introduce SketchSem3D, the first large-scale benchmark for generating 3D outdoor semantic scenes from abstract freehand sketches and pseudo-labeled annotations of satellite images. SketchSem3D includes two subsets, Sketch-based SemanticKITTI and Sketch-based KITTI-360 (containing LiDAR voxels along with their corresponding sketches and annotated satellite images), to enable standardized, rigorous, and diverse evaluations. We also propose Cylinder Mamba Diffusion (CymbaDiff) that significantly enhances spatial coherence in outdoor 3D scene generation. CymbaDiff imposes structured spatial ordering, explicitly captures cylindrical continuity and vertical hierarchy, and preserves both physical neighborhood relationships and global context within the generated scenes. Extensive experiments on SketchSem3D demonstrate that CymbaDiff achieves superior semantic consistency, spatial realism, and cross-dataset generalization. The code and dataset will be available at this https URL
Abstract (translated)
户外三维语义场景生成旨在为城市模拟和自动驾驶等应用创建现实且富含语义信息的环境。然而，这一领域的进步受到公开可用、标注完善的数据库缺失的限制。为此，我们推出了SketchSem3D，这是首个用于从抽象的手绘草图和卫星图像伪标签注释生成大规模三维户外语义场景的基准数据集。SketchSem3D包含两个子集：基于草图的SemanticKITTI和基于草图的KITTI-360（包括激光雷达体素及其对应的草图和标注过的卫星图像），以支持标准、严格且多样化的评估。此外，我们还提出了Cylinder Mamba Diffusion (CymbaDiff)，这一方法显著提升了户外三维场景生成的空间一致性。CymbaDiff施加了结构化空间顺序，并明确捕捉了圆柱连续性和垂直层次关系，在生成的场景中同时保留物理邻域关系和全局背景信息。在SketchSem3D上的广泛实验表明，CymbaDiff实现了更优的语义一致性和空间现实感，以及跨数据集的泛化能力。相关代码和数据集将在以下链接发布：[此网址] (请根据实际发布的地址替换该占位符)
URL
https://arxiv.org/abs/2510.13245
PDF
-
Hybrid Gaussian Splatting for Novel Urban View Synthesis
2025-10-14 09:09:13Mohamed Omran, Farhad Zanjani, Davide Abati, Jens Petersen, Amirhossein HabibianAbstract
This paper describes the Qualcomm AI Research solution to the RealADSim-NVS challenge, hosted at the RealADSim Workshop at ICCV 2025. The challenge concerns novel view synthesis in street scenes, and participants are required to generate, starting from car-centric frames captured during some training traversals, renders of the same urban environment as viewed from a different traversal (e.g. different street lane or car direction). Our solution is inspired by hybrid methods in scene generation and generative simulators merging gaussian splatting and diffusion models, and it is composed of two stages: First, we fit a 3D reconstruction of the scene and render novel views as seen from the target cameras. Then, we enhance the resulting frames with a dedicated single-step diffusion model. We discuss specific choices made in the initialization of gaussian primitives as well as the finetuning of the enhancer model and its training data curation. We report the performance of our model design and we ablate its components in terms of novel view quality as measured by PSNR, SSIM and LPIPS. On the public leaderboard reporting test results, our proposal reaches an aggregated score of 0.432, achieving the second place overall.
Abstract (translated)
这篇论文描述了高通AI研究为ICCV 2025举办的RealADSim Workshop中的RealADSim-NVS挑战提供的解决方案。该挑战涉及街景中视点新颖性合成问题，参赛者需要从训练过程中拍摄的以汽车为中心的画面出发，生成同一城市环境在不同视角下的渲染图（例如不同的街道车道或汽车方向）。我们的方案受到场景生成混合方法和结合高斯光斑技术和扩散模型的生成模拟器的启发，由两个阶段组成：首先，我们拟合出一个3D重建的场景，并从目标相机的角度重新渲染新颖视图；然后，我们使用专门设计的一步式扩散模型来增强所得帧。本文讨论了初始化高斯原语的具体选择以及细化增强模型及其训练数据整理方面的决策。我们报告了我们的模型的设计性能，并根据PSNR（峰值信噪比）、SSIM（结构相似性指数）和LPIPS（感知线性像差感知感知图像相似度）来衡量新颖视图质量，对其组件进行了消融研究。在公布的结果排行榜上，以测试结果为基础，我们方案的综合得分为0.432，取得了总成绩第二名的成绩。
URL
https://arxiv.org/abs/2510.12308
PDF
-
IL3D: A Large-Scale Indoor Layout Dataset for LLM-Driven 3D Scene Generation
2025-10-14 03:02:33Wenxu Zhou, Kaixuan Nie, Hang Du, Dong Yin, Wei Huang, Siqiang Guo, Xiaobo Zhang, Pengbo HuAbstract
In this study, we present IL3D, a large-scale dataset meticulously designed for large language model (LLM)-driven 3D scene generation, addressing the pressing demand for diverse, high-quality training data in indoor layout design. Comprising 27,816 indoor layouts across 18 prevalent room types and a library of 29,215 high-fidelity 3D object assets, IL3D is enriched with instance-level natural language annotations to support robust multimodal learning for vision-language tasks. We establish rigorous benchmarks to evaluate LLM-driven scene generation. Experimental results show that supervised fine-tuning (SFT) of LLMs on IL3D significantly improves generalization and surpasses the performance of SFT on other datasets. IL3D offers flexible multimodal data export capabilities, including point clouds, 3D bounding boxes, multiview images, depth maps, normal maps, and semantic masks, enabling seamless adaptation to various visual tasks. As a versatile and robust resource, IL3D significantly advances research in 3D scene generation and embodied intelligence, by providing high-fidelity scene data to support environment perception tasks of embodied agents.
Abstract (translated)
在这项研究中，我们介绍了IL3D，这是一个为大规模语言模型（LLM）驱动的三维场景生成精心设计的大规模数据集，旨在满足室内布局设计领域对多样化、高质量训练数据迫切需求。IL3D包含27,816个室内布局，涵盖了18种常见的房间类型，并拥有一个由29,215项高保真3D物体资产组成的库。此外，该数据集通过实例级别的自然语言注释进行了丰富，以支持视觉-语言任务中的稳健多模态学习。我们建立了严格的基准测试来评估LLM驱动的场景生成能力。实验结果显示，在IL3D上对LLMs进行监督微调（SFT）显著提高了泛化性能，并且优于在其他数据集上的SFT表现。 IL3D提供了灵活的多模态数据导出功能，包括点云、三维边界框、多视角图像、深度图、法线图和语义掩码，从而可以无缝地适应各种视觉任务。作为一个多功能且稳健的资源，IL3D通过提供高保真场景数据来支持具身智能体的环境感知任务，在三维场景生成和具身智能的研究方面取得了重大进展。
URL
https://arxiv.org/abs/2510.12095
PDF
-
EmboMatrix: A Scalable Training-Ground for Embodied Decision-Making
2025-10-14 02:26:52Zixing Lei, Sheng Yin, Yichen Xiong, Yuanzhuo Ding, Wenhao Huang, Yuxi Wei, Qingyao Xu, Yiming Li, Weixin Li, Yunhong Wang, Siheng ChenAbstract
Embodied decision-making enables agents to translate high-level goals into executable actions through continuous interactions within the physical world, forming a cornerstone of general-purpose embodied intelligence. Large language models (LLMs), with their general decision-making capabilities, offer a promising path to realize this potential; however, LLMs trained solely on language lack exposure to physical environments, limiting their true embodied understanding. To bridge this gap, we propose the concept of a training ground: a comprehensive infrastructure that provides task and scene simulation, embodied interaction, and feedback signals, offering a one-stop solution for LLM acquire genuine embodied decision-making skills. In this work, we present EmboMatrix, the first training ground of its kind, providing massive and diverse tasks with efficient simulation and precise rewards. EmboMatrix incorporates a series of novel techniques: a multi-agent data engine for large-scale task and scene generation, a distributed heterogeneous-hardware system for scalable simulation, and a multi-level reward architecture for precise supervision. Leveraging EmboMatrix, we cultivate EmboBrain, an LLM whose embodied decision-making abilities emerge from extensive embodied interactions. Experiments show that EmboBrain-7B surpasses the 671B DeepSeek-R1 baseline by 9.5\% on two challenging embodied decision-making benchmarks, demonstrating the power of interactive, environment-grounded learning for building truly intelligent embodied agents.
Abstract (translated)
嵌入式决策制定使代理能够通过与物理世界的持续互动将高层次的目标转化为可执行的动作，这是通用型嵌入智能的核心。大型语言模型（LLM）凭借其广泛的应用决策能力为实现这一潜力提供了可能路径；然而，仅接受语言训练的LLM缺乏对物理环境的实际接触，这限制了它们在真正意义上理解并适应物理世界的能力。为了弥合这种差距，我们提出了一种训练平台的概念：一个提供任务和场景模拟、嵌入式互动及反馈信号的全面基础设施，为LLM获取真正的嵌入式决策制定技能提供一站式解决方案。 在这项工作中，我们介绍了EmboMatrix——首个此类训练平台，它提供了大量且多样化的任务，并配备了高效仿真的功能和精确奖励机制。EmboMatrix整合了一系列新颖技术：一个多代理数据引擎用于大规模的任务和场景生成、一个分布式的异构硬件系统支持可扩展模拟以及一个多层级的奖励架构以实现精准监督。 通过利用EmboMatrix，我们培养出了EmboBrain——一种LLM，其嵌入式决策能力源于广泛的物理世界互动。实验表明，在两个具有挑战性的嵌入式决策制定基准测试中，EmboBrain-7B的表现比671B的DeepSeek-R1基线模型高出9.5%，这证明了基于环境交互的学习对于构建真正智能的嵌入式代理的重要性。
URL
https://arxiv.org/abs/2510.12072
PDF
-
REACT3D: Recovering Articulations for Interactive Physical 3D Scenes
2025-10-13 12:37:59Zhao Huang, Boyang Sun, Alexandros Delitzas, Jiaqi Chen, Marc PollefeysAbstract
Interactive 3D scenes are increasingly vital for embodied intelligence, yet existing datasets remain limited due to the labor-intensive process of annotating part segmentation, kinematic types, and motion trajectories. We present REACT3D, a scalable zero-shot framework that converts static 3D scenes into simulation-ready interactive replicas with consistent geometry, enabling direct use in diverse downstream tasks. Our contributions include: (i) openable-object detection and segmentation to extract candidate movable parts from static scenes, (ii) articulation estimation that infers joint types and motion parameters, (iii) hidden-geometry completion followed by interactive object assembly, and (iv) interactive scene integration in widely supported formats to ensure compatibility with standard simulation platforms. We achieve state-of-the-art performance on detection/segmentation and articulation metrics across diverse indoor scenes, demonstrating the effectiveness of our framework and providing a practical foundation for scalable interactive scene generation, thereby lowering the barrier to large-scale research on articulated scene understanding. Our project page is \textit{\hypersetup{urlcolor=black}\href{this https URL}{this http URL}}.
Abstract (translated)
交互式3D场景对于具身智能（embodied intelligence）的重要性日益增加，然而由于对部件分割、运动类型和轨迹标注的劳动密集型过程，现有的数据集仍然受限。我们提出了REACT3D，这是一个可扩展的零样本框架，能够将静态3D场景转换为具有连贯几何结构的交互式副本，可以直接用于各种下游任务中。我们的贡献包括： (i) 可开启物体检测和分割：从静态场景中提取候选移动部件； (ii) 轴组估计：推断关节类型和运动参数； (iii) 隐藏几何补全并进行交互对象组装； (iv) 在广泛支持的格式下实现交互式场景集成，以确保与标准仿真平台兼容。 我们在各种室内场景上的检测/分割和轴组度量上取得了最先进的性能，展示了我们框架的有效性，并为大规模交互式场景生成提供了一个实用的基础，从而降低了研究复杂连贯场景理解的大规模障碍。我们的项目页面位于 [此链接](this https URL)。
URL
https://arxiv.org/abs/2510.11340
PDF
-
From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries
2025-10-11 17:14:24Joy Hsu, Emily Jin, Jiajun Wu, Niloy J. MitraAbstract
Real-world scenes, such as those in ScanNet, are difficult to capture, with highly limited data available. Generating realistic scenes with varied object poses remains an open and challenging task. In this work, we propose FactoredScenes, a framework that synthesizes realistic 3D scenes by leveraging the underlying structure of rooms while learning the variation of object poses from lived-in scenes. We introduce a factored representation that decomposes scenes into hierarchically organized concepts of room programs and object poses. To encode structure, FactoredScenes learns a library of functions capturing reusable layout patterns from which scenes are drawn, then uses large language models to generate high-level programs, regularized by the learned library. To represent scene variations, FactoredScenes learns a program-conditioned model to hierarchically predict object poses, and retrieves and places 3D objects in a scene. We show that FactoredScenes generates realistic, real-world rooms that are difficult to distinguish from real ScanNet scenes.
Abstract (translated)
现实世界的场景，如ScanNet中的场景，难以捕捉，且可用的数据非常有限。生成具有多变物体姿态的逼真场景仍然是一个开放且具挑战性的任务。在这项工作中，我们提出了FactoredScenes框架，该框架通过利用房间的基础结构，并从日常生活场景中学习物体姿势的变化来合成逼真的3D场景。我们引入了一种分层的概念分解表示方法，将场景拆分为房间程序和物体姿态的组织化概念。 为了编码结构，FactoredScenes 学习了一个函数库，捕捉可重复使用的布局模式，然后利用大型语言模型生成高级程序，并通过学习到的库进行正则化处理。为了表示场景的变化，FactoredScenes 学习了一个条件于程序的模型来分层次预测物体姿态，并在场景中检索和放置3D对象。 我们展示了FactoredScenes能够生成难以与真实的ScanNet房间相区分的真实世界房间，证明了其有效性和创新性。
URL
https://arxiv.org/abs/2510.10292
PDF
-
MetaFind: Scene-Aware 3D Asset Retrieval for Coherent Metaverse Scene Generation
2025-10-05 06:37:26Zhenyu Pan, Yucheng Lu, Han LiuAbstract
We present MetaFind, a scene-aware tri-modal compositional retrieval framework designed to enhance scene generation in the metaverse by retrieving 3D assets from large-scale repositories. MetaFind addresses two core challenges: (i) inconsistent asset retrieval that overlooks spatial, semantic, and stylistic constraints, and (ii) the absence of a standardized retrieval paradigm specifically tailored for 3D asset retrieval, as existing approaches mainly rely on general-purpose 3D shape representation models. Our key innovation is a flexible retrieval mechanism that supports arbitrary combinations of text, image, and 3D modalities as queries, enhancing spatial reasoning and style consistency by jointly modeling object-level features (including appearance) and scene-level layout structures. Methodologically, MetaFind introduces a plug-and-play equivariant layout encoder ESSGNN that captures spatial relationships and object appearance features, ensuring retrieved 3D assets are contextually and stylistically coherent with the existing scene, regardless of coordinate frame transformations. The framework supports iterative scene construction by continuously adapting retrieval results to current scene updates. Empirical evaluations demonstrate the improved spatial and stylistic consistency of MetaFind in various retrieval tasks compared to baseline methods.
Abstract (translated)
我们介绍了MetaFind，这是一个场景感知的三模态组合检索框架，旨在通过从大规模存储库中检索3D资产来增强元宇宙中的场景生成。MetaFind解决了两个核心挑战：(i) 不一致的资产检索忽略了空间、语义和风格约束；(ii) 缺乏专门针对3D资产检索的标准检索范式，因为现有方法主要依赖于通用的3D形状表示模型。 我们的关键技术创新在于提供了一种灵活的检索机制，支持任意组合文本、图像和3D模态作为查询，通过联合建模对象级别的特征（包括外观）和场景级别布局结构来增强空间推理和风格一致性。从方法论上看，MetaFind引入了一个即插即用等变布局编码器ESSGNN，能够捕捉空间关系和物体的外观特征，并确保检索到的3D资产在上下文和风格上与现有场景一致，无论坐标系转换如何。 该框架支持迭代场景构建，通过持续地将检索结果适应于当前场景更新。经验评估表明，在各种检索任务中，MetaFind相较于基线方法显著提升了空间和风格一致性。
URL
https://arxiv.org/abs/2510.04057
PDF
-
Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft
2025-10-03 17:35:16Junchao Huang, Xinting Hu, Boyao Han, Shaoshuai Shi, Zhuotao Tian, Tianyu He, Li JiangAbstract
Autoregressive video diffusion models have proved effective for world modeling and interactive scene generation, with Minecraft gameplay as a representative application. To faithfully simulate play, a model must generate natural content while exploring new scenes and preserve spatial consistency when revisiting explored areas. Under limited computation budgets, it must compress and exploit historical cues within a finite context window, which exposes a trade-off: Temporal-only memory lacks long-term spatial consistency, whereas adding spatial memory strengthens consistency but may degrade new scene generation quality when the model over-relies on insufficient spatial context. We present Memory Forcing, a learning framework that pairs training protocols with a geometry-indexed spatial memory. Hybrid Training exposes distinct gameplay regimes, guiding the model to rely on temporal memory during exploration and incorporate spatial memory for revisits. Chained Forward Training extends autoregressive training with model rollouts, where chained predictions create larger pose variations and encourage reliance on spatial memory for maintaining consistency. Point-to-Frame Retrieval efficiently retrieves history by mapping currently visible points to their source frames, while Incremental 3D Reconstruction maintains and updates an explicit 3D cache. Extensive experiments demonstrate that Memory Forcing achieves superior long-term spatial consistency and generative quality across diverse environments, while maintaining computational efficiency for extended sequences.
Abstract (translated)
自回归视频扩散模型已被证明在世界建模和交互式场景生成方面非常有效，以《我的世界》游戏玩法作为代表性应用。为了真实地模拟游戏体验，一个模型必须在探索新场景时生成自然内容，并且当重新访问已探索的区域时保持空间一致性。在计算资源有限的情况下，该模型需要压缩并在有限的时间窗口内利用历史线索，这暴露了一个权衡：仅时间记忆缺乏长期的空间一致性，而添加空间记忆虽然可以增强一致性，但可能会因过度依赖不足的空间上下文而导致新场景生成质量下降。 我们提出了Memory Forcing（内存强制）框架，这是一个结合了几何索引空间内存的训练协议的学习框架。混合训练通过区分不同的游戏模式，引导模型在探索阶段依赖时间记忆，并在重访时引入空间记忆以保持一致性。链式前向训练则是在自回归训练的基础上添加了模型预测序列，在这种情况下，连续的预测会创建更大的姿态变化，并鼓励模型依赖空间内存来维持一致性。点到帧检索通过将当前可见点映射回其源帧来进行历史记录的有效检索，而增量3D重建则维护并更新显式的三维缓存。 广泛的实验表明，Memory Forcing框架在各种环境中实现了卓越的长期空间一致性和生成质量，同时保持了对延长序列计算效率的需求。
URL
https://arxiv.org/abs/2510.03198
PDF
-
Text-to-Scene with Large Reasoning Models
2025-09-30 11:08:11Fr\'ed\'eric Berdoz, Luca A. Lanzend\"orfer, Nick Tuninga, Roger WattenhoferAbstract
Prompt-driven scene synthesis allows users to generate complete 3D environments from textual descriptions. Current text-to-scene methods often struggle with complex geometries and object transformations, and tend to show weak adherence to complex instructions. We address these limitations by introducing Reason-3D, a text-to-scene model powered by large reasoning models (LRMs). Reason-3D integrates object retrieval using captions covering physical, functional, and contextual attributes. Reason-3D then places the selected objects based on implicit and explicit layout constraints, and refines their positions with collision-aware spatial reasoning. Evaluated on instructions ranging from simple to complex indoor configurations, Reason-3D significantly outperforms previous methods in human-rated visual fidelity, adherence to constraints, and asset retrieval quality. Beyond its contribution to the field of text-to-scene generation, our work showcases the advanced spatial reasoning abilities of modern LRMs. Additionally, we release the codebase to further the research in object retrieval and placement with LRMs.
Abstract (translated)
提示驱动的场景合成允许用户根据文本描述生成完整的3D环境。当前的文本到场景方法在处理复杂几何形状和对象变换时常常遇到困难，并且对复杂指令的遵循度较低。为了解决这些限制，我们引入了Reason-3D，这是一个由大型推理模型（LRMs）驱动的文本到场景模型。Reason-3D集成了基于描述物理属性、功能属性和上下文属性的对象检索功能。随后，它根据隐式和显式的布局约束放置选定对象，并通过碰撞感知的空间推理来优化它们的位置。 在从简单到复杂的室内配置指令上进行评估时，Reason-3D在人类评价的视觉逼真度、遵循约束的程度以及资产检索质量方面显著优于以前的方法。除了对文本到场景生成领域的贡献外，我们的工作还展示了现代LRMs先进的空间推理能力。此外，我们发布了代码库，以进一步推动LRMs在对象检索和放置方面的研究。
URL
https://arxiv.org/abs/2509.26091
PDF
-
M3DLayout: A Multi-Source Dataset of 3D Indoor Layouts and Structured Descriptions for 3D Generation
2025-09-28 08:16:08Yiheng Zhang, Zhuojiang Cai, Mingdao Wang, Meitong Guo, Tianxiao Li, Li Lin, Yuwang WangAbstract
In text-driven 3D scene generation, object layout serves as a crucial intermediate representation that bridges high-level language instructions with detailed geometric output. It not only provides a structural blueprint for ensuring physical plausibility but also supports semantic controllability and interactive editing. However, the learning capabilities of current 3D indoor layout generation models are constrained by the limited scale, diversity, and annotation quality of existing datasets. To address this, we introduce M3DLayout, a large-scale, multi-source dataset for 3D indoor layout generation. M3DLayout comprises 15,080 layouts and over 258k object instances, integrating three distinct sources: real-world scans, professional CAD designs, and procedurally generated scenes. Each layout is paired with detailed structured text describing global scene summaries, relational placements of large furniture, and fine-grained arrangements of smaller items. This diverse and richly annotated resource enables models to learn complex spatial and semantic patterns across a wide variety of indoor environments. To assess the potential of M3DLayout, we establish a benchmark using a text-conditioned diffusion model. Experimental results demonstrate that our dataset provides a solid foundation for training layout generation models. Its multi-source composition enhances diversity, notably through the Inf3DLayout subset which provides rich small-object information, enabling the generation of more complex and detailed scenes. We hope that M3DLayout can serve as a valuable resource for advancing research in text-driven 3D scene synthesis.
Abstract (translated)
在基于文本驱动的三维场景生成中，物体布局作为一种关键的中间表示形式，连接了高层次的语言指令和详细的几何输出。它不仅提供了确保物理合理性的结构蓝图，还支持语义可控性和交互式编辑。然而，当前三维室内布局生成模型的学习能力受到现有数据集规模有限、多样性不足以及标注质量不高的限制。为了解决这些问题，我们引入了M3DLayout，这是一个大规模的多源数据集，用于三维室内布局生成。M3DLayout包含15,080个布局和超过258k个物体实例，整合了三个不同的来源：真实世界的扫描、专业CAD设计以及程序化生成场景。每个布局都配有一份详细的结构化文本描述，包括全局场景概要、大型家具的相对位置关系以及小型物品的精细布置安排。这种多样性和丰富标注的数据资源使模型能够学习到复杂的空间和语义模式，在各种室内环境中具有广泛的适用性。 为了评估M3DLayout的潜力，我们使用基于条件文本扩散模型建立了一个基准测试平台。实验结果表明，我们的数据集为训练布局生成模型提供了坚实的基础。其多源构成增强了多样性，特别是Inf3DLayout子集提供的丰富的小物体信息，使更复杂和详细的场景生成成为可能。我们希望M3DLayout能够作为一个有价值的资源，推动基于文本驱动的三维场景合成研究的进步。
URL
https://arxiv.org/abs/2509.23728
PDF
