Scene_Generation
-
Vision Technologies with Applications in Traffic Surveillance Systems: A Holistic Survey
2024-11-30 04:17:56Wei Zhou, Lei Zhao, Runyu Zhang, Yifan Cui, Hongpu Huang, Kun Qie, Chen WangAbstract
Traffic Surveillance Systems (TSS) have become increasingly crucial in modern intelligent transportation systems, with vision-based technologies playing a central role for scene perception and understanding. While existing surveys typically focus on isolated aspects of TSS, a comprehensive analysis bridging low-level and high-level perception tasks, particularly considering emerging technologies, remains lacking. This paper presents a systematic review of vision-based technologies in TSS, examining both low-level perception tasks (object detection, classification, and tracking) and high-level perception applications (parameter estimation, anomaly detection, and behavior understanding). Specifically, we first provide a detailed methodological categorization and comprehensive performance evaluation for each task. Our investigation reveals five fundamental limitations in current TSS: perceptual data degradation in complex scenarios, data-driven learning constraints, semantic understanding gaps, sensing coverage limitations and computational resource demands. To address these challenges, we systematically analyze five categories of potential solutions: advanced perception enhancement, efficient learning paradigms, knowledge-enhanced understanding, cooperative sensing frameworks and efficient computing frameworks. Furthermore, we evaluate the transformative potential of foundation models in TSS, demonstrating their unique capabilities in zero-shot learning, semantic understanding, and scene generation. This review provides a unified framework bridging low-level and high-level perception tasks, systematically analyzes current limitations and solutions, and presents a structured roadmap for integrating emerging technologies, particularly foundation models, to enhance TSS capabilities.
Abstract (translated)
URL
https://arxiv.org/abs/2412.00348
PDF
-
Graph Canvas for Controllable 3D Scene Generation
2024-11-27 12:41:23Libin Liu, Shen Chen, Sen Jia, Jingzhe Shi, Zhongyu Jiang, Can Jin, Wu Zongkai, Jenq-Neng Hwang, Lei LiAbstract
Spatial intelligence is foundational to AI systems that interact with the physical world, particularly in 3D scene generation and spatial comprehension. Current methodologies for 3D scene generation often rely heavily on predefined datasets, and struggle to adapt dynamically to changing spatial relationships. In this paper, we introduce \textbf{GraphCanvas3D}, a programmable, extensible, and adaptable framework for controllable 3D scene generation. Leveraging in-context learning, GraphCanvas3D enables dynamic adaptability without the need for retraining, supporting flexible and customizable scene creation. Our framework employs hierarchical, graph-driven scene descriptions, representing spatial elements as graph nodes and establishing coherent relationships among objects in 3D environments. Unlike conventional approaches, which are constrained in adaptability and often require predefined input masks or retraining for modifications, GraphCanvas3D allows for seamless object manipulation and scene adjustments on the fly. Additionally, GraphCanvas3D supports 4D scene generation, incorporating temporal dynamics to model changes over time. Experimental results and user studies demonstrate that GraphCanvas3D enhances usability, flexibility, and adaptability for scene generation. Our code and models are available on the project website: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2412.00091
PDF
-
ROOT: VLM based System for Indoor Scene Understanding and Beyond
2024-11-24 04:51:24Yonghui Wang, Shi-Yong Chen, Zhenxing Zhou, Siyi Li, Haoran Li, Wengang Zhou, Houqiang LiAbstract
Recently, Vision Language Models (VLMs) have experienced significant advancements, yet these models still face challenges in spatial hierarchical reasoning within indoor scenes. In this study, we introduce ROOT, a VLM-based system designed to enhance the analysis of indoor scenes. Specifically, we first develop an iterative object perception algorithm using GPT-4V to detect object entities within indoor scenes. This is followed by employing vision foundation models to acquire additional meta-information about the scene, such as bounding boxes. Building on this foundational data, we propose a specialized VLM, SceneVLM, which is capable of generating spatial hierarchical scene graphs and providing distance information for objects within indoor environments. This information enhances our understanding of the spatial arrangement of indoor scenes. To train our SceneVLM, we collect over 610,000 images from various public indoor datasets and implement a scene data generation pipeline with a semi-automated technique to establish relationships and estimate distances among indoor objects. By utilizing this enriched data, we conduct various training recipes and finish SceneVLM. Our experiments demonstrate that \rootname facilitates indoor scene understanding and proves effective in diverse downstream applications, such as 3D scene generation and embodied AI. The code will be released at \url{this https URL}.
Abstract (translated)
最近，视觉语言模型（VLMs）取得了显著的进步，但这些模型在处理室内场景中的空间层次推理方面仍面临挑战。在这项研究中，我们介绍了ROOT——一个基于VLM的系统，旨在提高对室内场景的分析能力。具体来说，首先我们开发了一个迭代的对象感知算法，使用GPT-4V来检测室内场景中的对象实体。接着，利用视觉基础模型获取关于该场景的额外元信息，如边界框等。在此基础上，我们提出了一种专门的VLM——SceneVLM，它能够生成空间层次化的场景图，并提供室内环境中物体之间的距离信息。这些信息提升了我们对室内场景空间布局的理解。为了训练我们的SceneVLM，我们从各种公共的室内数据集中收集了超过610,000张图像，并采用半自动技术实现了一个场景数据生成管道来建立物体间的关系并估计它们的距离。通过利用丰富的数据，我们进行了多种训练方法并完成了SceneVLM的构建。实验表明，\rootname促进了对室内场景的理解，在各种下游应用中（如3D场景生成和具身AI）证明了其有效性。代码将在\url{这个 https URL}公开发布。
URL
https://arxiv.org/abs/2411.15714
PDF
-
What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generation
2024-11-23 03:40:25Zuyao Chen, Jinlin Wu, Zhen Lei, Chang Wen ChenAbstract
While text-to-image generation has been extensively studied, generating images from scene graphs remains relatively underexplored, primarily due to challenges in accurately modeling spatial relationships and object interactions. To fill this gap, we introduce Scene-Bench, a comprehensive benchmark designed to evaluate and enhance the factual consistency in generating natural scenes. Scene-Bench comprises MegaSG, a large-scale dataset of one million images annotated with scene graphs, facilitating the training and fair comparison of models across diverse and complex scenes. Additionally, we propose SGScore, a novel evaluation metric that leverages chain-of-thought reasoning capabilities of multimodal large language models (LLMs) to assess both object presence and relationship accuracy, offering a more effective measure of factual consistency than traditional metrics like FID and CLIPScore. Building upon this evaluation framework, we develop a scene graph feedback pipeline that iteratively refines generated images by identifying and correcting discrepancies between the scene graph and the image. Extensive experiments demonstrate that Scene-Bench provides a more comprehensive and effective evaluation framework compared to existing benchmarks, particularly for complex scene generation. Furthermore, our feedback strategy significantly enhances the factual consistency of image generation models, advancing the field of controllable image generation.
Abstract (translated)
尽管文本到图像的生成已经得到了广泛的研究，但从场景图生成图像仍然相对较少被探索，主要是因为准确建模空间关系和对象互动存在挑战。为了填补这一空白，我们引入了Scene-Bench，这是一个旨在评估并提高自然场景生成中事实一致性的全面基准测试。Scene-Bench 包含 MegaSG，一个由一百万张带有场景图注释的大型数据集，有助于在多样且复杂的场景中训练和公平比较模型。此外，我们提出了 SGScore，这是一种新颖的评价指标，利用多模态大语言模型（LLMs）的链式思考推理能力来评估对象存在及其关系的准确性，相比传统的FID和CLIPScore等度量标准，提供了更有效的事实一致性衡量方法。基于这一评估框架，我们开发了一个场景图反馈管道，通过识别并修正图像与场景图之间的差异来迭代地改进生成的图像。广泛的实验表明，Scene-Bench 提供了比现有基准测试更加全面且有效的评估框架，特别是在复杂场景生成方面。此外，我们的反馈策略显著提升了图像生成模型的事实一致性，推动了可控图像生成领域的进步。
URL
https://arxiv.org/abs/2411.15435
PDF
-
Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation
2024-11-21 18:21:24Yuanhao Cai, He Zhang, Kai Zhang, Yixun Liang, Mengwei Ren, Fujun Luan, Qing Liu, Soo Ye Kim, Jianming Zhang, Zhifei Zhang, Yuqian Zhou, Zhe Lin, Alan YuilleAbstract
Existing feed-forward image-to-3D methods mainly rely on 2D multi-view diffusion models that cannot guarantee 3D consistency. These methods easily collapse when changing the prompt view direction and mainly handle object-centric prompt images. In this paper, we propose a novel single-stage 3D diffusion model, DiffusionGS, for object and scene generation from a single view. DiffusionGS directly outputs 3D Gaussian point clouds at each timestep to enforce view consistency and allow the model to generate robustly given prompt views of any directions, beyond object-centric inputs. Plus, to improve the capability and generalization ability of DiffusionGS, we scale up 3D training data by developing a scene-object mixed training strategy. Experiments show that our method enjoys better generation quality (2.20 dB higher in PSNR and 23.25 lower in FID) and over 5x faster speed (~6s on an A100 GPU) than SOTA methods. The user study and text-to-3D applications also reveals the practical values of our method. Our Project page at this https URL shows the video and interactive generation results.
Abstract (translated)
现有的前馈图像到三维方法主要依赖于二维多视角扩散模型，这些模型无法保证三维一致性。当改变提示视图方向时，这些方法容易失效，并且主要处理以物体为中心的提示图像。在本文中，我们提出了一种新的单阶段三维扩散模型DiffusionGS，用于从单一视图生成对象和场景。DiffusionGS在每个时间步直接输出3D高斯点云，以强制执行视角一致性并允许模型稳健地生成任何方向上的提示视图，而不仅仅限于以物体为中心的输入。此外，为了提高DiffusionGS的能力和泛化能力，我们通过开发一种场景-对象混合训练策略来扩展三维训练数据规模。实验表明，我们的方法在生成质量（PSNR高2.20 dB，FID低23.25）上优于现有最佳方法，并且速度超过其五倍（约6秒，在A100 GPU上）。用户研究和文本到三维应用也揭示了我们方法的实际价值。我们的项目页面在此链接中展示了视频和交互生成结果：[https URL]。
URL
https://arxiv.org/abs/2411.14384
PDF
-
SSEditor: Controllable Mask-to-Scene Generation with Diffusion Model
2024-11-19 07:19:05Haowen Zheng, Yanyan LiangAbstract
Recent advancements in 3D diffusion-based semantic scene generation have gained attention. However, existing methods rely on unconditional generation and require multiple resampling steps when editing scenes, which significantly limits their controllability and flexibility. To this end, we propose SSEditor, a controllable Semantic Scene Editor that can generate specified target categories without multiple-step resampling. SSEditor employs a two-stage diffusion-based framework: (1) a 3D scene autoencoder is trained to obtain latent triplane features, and (2) a mask-conditional diffusion model is trained for customizable 3D semantic scene generation. In the second stage, we introduce a geometric-semantic fusion module that enhance the model's ability to learn geometric and semantic information. This ensures that objects are generated with correct positions, sizes, and categories. Extensive experiments on SemanticKITTI and CarlaSC demonstrate that SSEditor outperforms previous approaches in terms of controllability and flexibility in target generation, as well as the quality of semantic scene generation and reconstruction. More importantly, experiments on the unseen Occ-3D Waymo dataset show that SSEditor is capable of generating novel urban scenes, enabling the rapid construction of 3D scenes.
Abstract (translated)
最近，基于3D扩散的语义场景生成技术引起了广泛关注。然而，现有的方法依赖于无条件生成，并且在编辑场景时需要多次重采样步骤，这极大地限制了它们的可控性和灵活性。为此，我们提出了SSEditor，一个可控制的语义场景编辑器，它可以生成指定的目标类别而无需多步重采样。SSEditor采用了基于扩散的两阶段框架：（1）训练3D场景自编码器以获得潜在的三平面特征；（2）训练带有掩码条件的扩散模型进行定制化的3D语义场景生成。在第二阶段，我们引入了一个几何-语义融合模块，增强了模型学习几何和语义信息的能力。这确保了生成的对象具有正确的位置、大小和类别。在SemanticKITTI和CarlaSC上的广泛实验表明，SSEditor在目标生成的可控性和灵活性方面以及语义场景生成和重建的质量上优于先前的方法。更重要的是，在未见过的Occ-3D Waymo数据集上的实验显示，SSEditor能够生成新颖的城市场景，使3D场景的快速构建成为可能。
URL
https://arxiv.org/abs/2411.12290
PDF
-
Planning by Simulation: Motion Planning with Learning-based Parallel Scenario Prediction for Autonomous Driving
2024-11-15 02:18:35Tian Niu, Kaizhao Zhang, Zhongxue Gan, Wenchao DingAbstract
Planning safe trajectories for autonomous vehicles is essential for operational safety but remains extremely challenging due to the complex interactions among traffic participants. Recent autonomous driving frameworks have focused on improving prediction accuracy to explicitly model these interactions. However, some methods overlook the significant influence of the ego vehicle's planning on the possible trajectories of other agents, which can alter prediction accuracy and lead to unsafe planning decisions. In this paper, we propose a novel motion Planning approach by Simulation with learning-based parallel scenario prediction (PS). PS deduces predictions iteratively based on Monte Carlo Tree Search (MCTS), jointly inferring scenarios that cooperate with the ego vehicle's planning set. Our method simulates possible scenes and calculates their costs after the ego vehicle executes potential actions. To balance and prune unreasonable actions and scenarios, we adopt MCTS as the foundation to explore possible future interactions encoded within the prediction network. Moreover, the query-centric trajectory prediction streamlines our scene generation, enabling a sophisticated framework that captures the mutual influence between other agents' predictions and the ego vehicle's planning. We evaluate our framework on the Argoverse 2 dataset, and the results demonstrate that our approach effectively achieves parallel ego vehicle planning.
Abstract (translated)
规划自动驾驶车辆的安全轨迹对于操作安全性至关重要，但由于交通参与者之间的复杂交互作用，这一过程仍然极具挑战性。最近的自主驾驶框架专注于提高预测准确性以明确建模这些交互作用。然而，一些方法忽视了自车规划对其他代理可能轨迹的重大影响，这可能会改变预测准确性并导致不安全的规划决策。在本文中，我们提出了一种基于仿真与学习型平行场景预测（PS）的新运动规划方法。PS 基于蒙特卡洛树搜索（MCTS），迭代地推导出与自车规划集合作用的场景推测。我们的方法模拟可能的情景，并计算自车执行潜在动作后的成本。为了平衡和修剪不合理的动作和情景，我们采用 MCTS 作为基础来探索预测网络中编码的可能未来交互作用。此外，以查询为中心的轨迹预测简化了我们的场景生成过程，使我们能够构建一个复杂的框架，捕捉其他代理预测与自车规划之间的相互影响。我们在 Argoverse 2 数据集上评估了我们的框架，结果表明，我们的方法有效地实现了平行的自车规划。
URL
https://arxiv.org/abs/2411.09887
PDF
-
Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting
2024-11-14 22:15:48Yian Wang, Xiaowen Qiu, Jiageng Liu, Zhehuan Chen, Jiting Cai, Yufei Wang, Tsun-Hsuan Wang, Zhou Xian, Chuang GanAbstract
Creating large-scale interactive 3D environments is essential for the development of Robotics and Embodied AI research. Current methods, including manual design, procedural generation, diffusion-based scene generation, and large language model (LLM) guided scene design, are hindered by limitations such as excessive human effort, reliance on predefined rules or training datasets, and limited 3D spatial reasoning ability. Since pre-trained 2D image generative models better capture scene and object configuration than LLMs, we address these challenges by introducing Architect, a generative framework that creates complex and realistic 3D embodied environments leveraging diffusion-based 2D image inpainting. In detail, we utilize foundation visual perception models to obtain each generated object from the image and leverage pre-trained depth estimation models to lift the generated 2D image to 3D space. Our pipeline is further extended to a hierarchical and iterative inpainting process to continuously generate placement of large furniture and small objects to enrich the scene. This iterative structure brings the flexibility for our method to generate or refine scenes from various starting points, such as text, floor plans, or pre-arranged environments.
Abstract (translated)
创建大规模交互式3D环境对于机器人技术和具身人工智能研究的发展至关重要。当前的方法，包括手动设计、程序生成、基于扩散的场景生成以及大型语言模型（LLM）引导的场景设计，都受到过度依赖人力投入、对预定义规则或训练数据集的依赖性以及有限的3D空间推理能力等限制的影响。鉴于预先训练的2D图像生成模型比LLM更能捕捉到场景和物体配置的特点，我们通过引入Architect解决了这些挑战——这是一种利用基于扩散的2D图像修复来创建复杂且逼真的3D具身环境的生成框架。具体来说，我们使用基础视觉感知模型从图像中获取每个生成的对象，并借助预先训练好的深度估计模型将生成的2D图像提升到3D空间。我们的流程进一步扩展为一个分层和迭代的修复过程，以持续生成大型家具和小型物体的位置来丰富场景。这种迭代结构带来了灵活性，使我们的方法能够从各种起点（如文本、平面图或预先布置的环境）开始生成或细化场景。
URL
https://arxiv.org/abs/2411.09823
PDF
-
DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion
2024-11-07 18:07:31Wenqiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, Yikai WangAbstract
In this paper, we introduce \textbf{DimensionX}, a framework designed to generate photorealistic 3D and 4D scenes from just a single image with video diffusion. Our approach begins with the insight that both the spatial structure of a 3D scene and the temporal evolution of a 4D scene can be effectively represented through sequences of video frames. While recent video diffusion models have shown remarkable success in producing vivid visuals, they face limitations in directly recovering 3D/4D scenes due to limited spatial and temporal controllability during generation. To overcome this, we propose ST-Director, which decouples spatial and temporal factors in video diffusion by learning dimension-aware LoRAs from dimension-variant data. This controllable video diffusion approach enables precise manipulation of spatial structure and temporal dynamics, allowing us to reconstruct both 3D and 4D representations from sequential frames with the combination of spatial and temporal dimensions. Additionally, to bridge the gap between generated videos and real-world scenes, we introduce a trajectory-aware mechanism for 3D generation and an identity-preserving denoising strategy for 4D generation. Extensive experiments on various real-world and synthetic datasets demonstrate that DimensionX achieves superior results in controllable video generation, as well as in 3D and 4D scene generation, compared with previous methods.
Abstract (translated)
在这篇论文中，我们介绍了**DimensionX**框架，该框架旨在通过仅使用带有视频扩散的单个图像来生成逼真的3D和4D场景。我们的方法基于这样的见解：即一个3D场景的空间结构以及一个4D场景的时间演变可以通过一系列视频帧有效地表示出来。尽管近期的视频扩散模型在产生生动的视觉效果方面取得了显著成功，但由于在生成过程中存在空间和时间控制方面的限制，它们直接恢复3D/4D场景的能力受到了限制。为克服这一问题，我们提出了ST-Director，它通过从维度变化的数据中学习维度感知LoRAs（低秩适配器），从而将视频扩散中的空间因素与时间因素解耦。这种可控的视频扩散方法能够精确操控空间结构和时间动态，使我们能结合空间和时间维度从连续帧重建3D和4D表示。此外，为了弥合生成视频与现实场景之间的差距，我们引入了针对3D生成的轨迹感知机制以及用于4D生成的身份保持去噪策略。在各种真实世界和合成数据集上的广泛实验表明，DimensionX相较于之前的方法，在可控视频生成以及3D和4D场景生成方面取得了更优的结果。
URL
https://arxiv.org/abs/2411.04928
PDF
-
DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion
2024-10-31 17:57:02Weicai Ye, Chenhao Ji, Zheng Chen, Junyao Gao, Xiaoshui Huang, Song-Hai Zhang, Wanli Ouyang, Tong He, Cairong Zhao, Guofeng ZhangAbstract
Diffusion-based methods have achieved remarkable achievements in 2D image or 3D object generation, however, the generation of 3D scenes and even $360^{\circ}$ images remains constrained, due to the limited number of scene datasets, the complexity of 3D scenes themselves, and the difficulty of generating consistent multi-view images. To address these issues, we first establish a large-scale panoramic video-text dataset containing millions of consecutive panoramic keyframes with corresponding panoramic depths, camera poses, and text descriptions. Then, we propose a novel text-driven panoramic generation framework, termed DiffPano, to achieve scalable, consistent, and diverse panoramic scene generation. Specifically, benefiting from the powerful generative capabilities of stable diffusion, we fine-tune a single-view text-to-panorama diffusion model with LoRA on the established panoramic video-text dataset. We further design a spherical epipolar-aware multi-view diffusion model to ensure the multi-view consistency of the generated panoramic images. Extensive experiments demonstrate that DiffPano can generate scalable, consistent, and diverse panoramic images with given unseen text descriptions and camera poses.
Abstract (translated)
基于扩散的方法在2D图像或3D物体生成方面取得了显著成就，然而，由于场景数据集的数量有限、3D场景本身的复杂性以及生成一致的多视角图像的难度，3D场景和甚至$360^{\circ}$图像的生成仍然受到限制。为了解决这些问题，我们首先建立了一个大规模全景视频文本数据集，包含数百万个连续的全景关键帧及其对应的全景深度、相机姿态和文本描述。然后，我们提出了一种新颖的以文本驱动的全景生成框架，称为DiffPano，旨在实现可扩展、一致且多样化的全景场景生成。具体来说，得益于稳定扩散的强大生成能力，我们在建立的全景视频文本数据集上使用LoRA对单视角文本到全景的扩散模型进行了微调。我们还设计了一种球面视差感知多视角扩散模型，以确保生成的全景图像具有多视角一致性。大量的实验表明，DiffPano可以生成可扩展、一致且多样化的全景图像，并能根据给定的未见过的文本描述和相机姿态进行生成。
URL
https://arxiv.org/abs/2410.24203
PDF
-
SceneGenAgent: Precise Industrial Scene Generation with Coding Agent
2024-10-29 10:01:40Xiao Xia, Dan Zhang, Zibo Liao, Zhenyu Hou, Tianrui Sun, Jing Li, Ling Fu, Yuxiao DongAbstract
The modeling of industrial scenes is essential for simulations in industrial manufacturing. While large language models (LLMs) have shown significant progress in generating general 3D scenes from textual descriptions, generating industrial scenes with LLMs poses a unique challenge due to their demand for precise measurements and positioning, requiring complex planning over spatial arrangement. To address this challenge, we introduce SceneGenAgent, an LLM-based agent for generating industrial scenes through C# code. SceneGenAgent ensures precise layout planning through a structured and calculable format, layout verification, and iterative refinement to meet the quantitative requirements of industrial scenarios. Experiment results demonstrate that LLMs powered by SceneGenAgent exceed their original performance, reaching up to 81.0% success rate in real-world industrial scene generation tasks and effectively meeting most scene generation requirements. To further enhance accessibility, we construct SceneInstruct, a dataset designed for fine-tuning open-source LLMs to integrate into SceneGenAgent. Experiments show that fine-tuning open-source LLMs on SceneInstruct yields significant performance improvements, with Llama3.1-70B approaching the capabilities of GPT-4o. Our code and data are available at this https URL .
Abstract (translated)
工业场景的建模对于工业制造中的模拟至关重要。虽然大型语言模型（LLMs）在根据文本描述生成通用的3D场景方面已经取得了显著进展，但使用LLMs生成工业场景却面临着独特的挑战，这主要是由于其对精确测量和定位的需求，需要进行复杂的空间布局规划。为了解决这一问题，我们引入了SceneGenAgent，这是一个基于LLM的代理程序，通过C#代码来生成工业场景。SceneGenAgent通过结构化且可计算的格式确保精准的布局规划，并通过布局验证及迭代优化来满足工业场景的定量要求。实验结果表明，由SceneGenAgent驱动的LLMs其性能超过了原有的水平，在真实世界中的工业场景生成任务中达到了高达81.0%的成功率，并有效地满足了大部分场景生成的要求。为了进一步提高可访问性，我们构建了SceneInstruct数据集，旨在对开源LLMs进行微调以集成到SceneGenAgent中。实验表明，在SceneInstruct上对开源LLMs进行微调能够显著提升性能，Llama3.1-70B接近达到了GPT-4o的能力水平。我们的代码和数据可以在这个 https URL 中获取。
URL
https://arxiv.org/abs/2410.21909
PDF
-
CompGS: Unleashing 2D Compositionality for Compositional Text-to-3D via Dynamically Optimizing 3D Gaussians
2024-10-28 04:35:14Chongjian Ge, Chenfeng Xu, Yuanfeng Ji, Chensheng Peng, Masayoshi Tomizuka, Ping Luo, Mingyu Ding, Varun Jampani, Wei ZhanAbstract
Recent breakthroughs in text-guided image generation have significantly advanced the field of 3D generation. While generating a single high-quality 3D object is now feasible, generating multiple objects with reasonable interactions within a 3D space, a.k.a. compositional 3D generation, presents substantial challenges. This paper introduces CompGS, a novel generative framework that employs 3D Gaussian Splatting (GS) for efficient, compositional text-to-3D content generation. To achieve this goal, two core designs are proposed: (1) 3D Gaussians Initialization with 2D compositionality: We transfer the well-established 2D compositionality to initialize the Gaussian parameters on an entity-by-entity basis, ensuring both consistent 3D priors for each entity and reasonable interactions among multiple entities; (2) Dynamic Optimization: We propose a dynamic strategy to optimize 3D Gaussians using Score Distillation Sampling (SDS) loss. CompGS first automatically decomposes 3D Gaussians into distinct entity parts, enabling optimization at both the entity and composition levels. Additionally, CompGS optimizes across objects of varying scales by dynamically adjusting the spatial parameters of each entity, enhancing the generation of fine-grained details, particularly in smaller entities. Qualitative comparisons and quantitative evaluations on T3Bench demonstrate the effectiveness of CompGS in generating compositional 3D objects with superior image quality and semantic alignment over existing methods. CompGS can also be easily extended to controllable 3D editing, facilitating scene generation. We hope CompGS will provide new insights to the compositional 3D generation. Project page: this https URL.
Abstract (translated)
近期，文本引导的图像生成领域取得了显著突破，大大推动了三维生成技术的发展。虽然现在生成单个高质量的3D对象已经成为可能，但在三维空间内生成具有合理互动的多个对象，即组合式3D生成，仍面临巨大挑战。本文介绍了一种新型生成框架CompGS，它利用3D高斯喷绘（GS）实现高效的、基于文本到3D内容的组合生成。为达到这一目标，提出了两个核心设计：(1) 基于2D组合性的3D高斯初始化：我们借鉴成熟的2D组合性原理，在实体级别上初始化高斯参数，确保每个实体都有一致的3D先验，并且多个实体之间存在合理的互动；(2) 动态优化：提出了一种动态策略，使用得分蒸馏采样（SDS）损失来优化3D高斯。CompGS首先自动将3D高斯分解成不同的实体部分，从而能够在实体和组合两个层面上进行优化。此外，通过动态调整每个实体的空间参数，CompGS可以跨不同尺度的对象进行优化，增强细节生成能力，特别是在较小的实体上。T3Bench上的定性和定量评估表明，与现有方法相比，CompGS在生成具有优越图像质量和语义对齐的组合式3D对象方面表现出色。CompGS还能够轻松扩展到可控的3D编辑，促进场景生成。我们希望CompGS能为组合式3D生成提供新的见解。项目页面：[此处链接]
URL
https://arxiv.org/abs/2410.20723
PDF
-
SCube: Instant Large-Scale Scene Reconstruction using VoxSplats
2024-10-26 00:52:46Xuanchi Ren, Yifan Lu, Hanxue Liang, Zhangjie Wu, Huan Ling, Mike Chen, Sanja Fidler, Francis Williams, Jiahui HuangAbstract
We present SCube, a novel method for reconstructing large-scale 3D scenes (geometry, appearance, and semantics) from a sparse set of posed images. Our method encodes reconstructed scenes using a novel representation VoxSplat, which is a set of 3D Gaussians supported on a high-resolution sparse-voxel scaffold. To reconstruct a VoxSplat from images, we employ a hierarchical voxel latent diffusion model conditioned on the input images followed by a feedforward appearance prediction model. The diffusion model generates high-resolution grids progressively in a coarse-to-fine manner, and the appearance network predicts a set of Gaussians within each voxel. From as few as 3 non-overlapping input images, SCube can generate millions of Gaussians with a 1024^3 voxel grid spanning hundreds of meters in 20 seconds. Past works tackling scene reconstruction from images either rely on per-scene optimization and fail to reconstruct the scene away from input views (thus requiring dense view coverage as input) or leverage geometric priors based on low-resolution models, which produce blurry results. In contrast, SCube leverages high-resolution sparse networks and produces sharp outputs from few views. We show the superiority of SCube compared to prior art using the Waymo self-driving dataset on 3D reconstruction and demonstrate its applications, such as LiDAR simulation and text-to-scene generation.
Abstract (translated)
我们提出了SCube，这是一种全新的方法，可以从一组稀疏的已定位图像中重建大规模的3D场景（包括几何结构、外观和语义）。我们的方法使用了一种新颖的表示形式VoxSplat来编码重建后的场景，这种表示形式是一组基于高分辨率稀疏体素框架支持的三维高斯分布。为了从图像中重建一个VoxSplat，我们采用了一个分层的体素潜在扩散模型，该模型依赖于输入图像，并随后使用前馈外观预测模型进行处理。扩散模型以由粗到细的方式逐步生成高分辨率网格，而外观网络则在每个体素内预测一组高斯分布。SCube可以从少至3个不重叠的输入图像中，在20秒内生成数百万个高斯分布在1024^3的体素网格上，该网格横跨数百米。 以往处理从图像进行场景重建的工作要么依赖于针对每个场景的优化，并且无法在远离输入视图的地方重构场景（因此需要密集视图覆盖作为输入），要么基于低分辨率模型使用几何先验，这会产生模糊的结果。相比之下，SCube利用高分辨率稀疏网络，能够从少量视图中生成清晰的结果。 我们通过Waymo自动驾驶数据集上的3D重建任务展示了SCube相较于以往技术的优势，并演示了它的应用，如激光雷达模拟和文本到场景的生成。
URL
https://arxiv.org/abs/2410.20030
PDF
-
Learning Global Object-Centric Representations via Disentangled Slot Attention
2024-10-24 14:57:00Tonglin Chen, Yinxuan Huang, Zhimeng Shen, Jinghao Huang, Bin Li, Xiangyang XueAbstract
Humans can discern scene-independent features of objects across various environments, allowing them to swiftly identify objects amidst changing factors such as lighting, perspective, size, and position and imagine the complete images of the same object in diverse settings. Existing object-centric learning methods only extract scene-dependent object-centric representations, lacking the ability to identify the same object across scenes as humans. Moreover, some existing methods discard the individual object generation capabilities to handle complex scenes. This paper introduces a novel object-centric learning method to empower AI systems with human-like capabilities to identify objects across scenes and generate diverse scenes containing specific objects by learning a set of global object-centric representations. To learn the global object-centric representations that encapsulate globally invariant attributes of objects (i.e., the complete appearance and shape), this paper designs a Disentangled Slot Attention module to convert the scene features into scene-dependent attributes (such as scale, position and orientation) and scene-independent representations (i.e., appearance and shape). Experimental results substantiate the efficacy of the proposed method, demonstrating remarkable proficiency in global object-centric representation learning, object identification, scene generation with specific objects and scene decomposition.
Abstract (translated)
人类能够在各种环境中识别出物体的环境无关特征，使他们能够迅速在光照、视角、大小和位置等变化因素中辨别出物体，并想象同一物体在不同场景中的完整图像。现有的以对象为中心的学习方法仅提取了依赖于特定场景的对象表示，缺乏像人类一样跨场景识别同一对象的能力。此外，某些现有方法放弃了生成单个对象的能力来处理复杂的场景。本文介绍了一种新的以对象为中心的学习方法，旨在赋予AI系统类似人类的跨场景识别物体和通过学习一组全局对象中心表示来生成包含特定对象的各种场景的能力。为了学习封装了对象全球不变属性（即完整外观和形状）的全局对象中心表示，本文设计了一个解缠插槽注意力模块将场景特征转换为依赖于场景的属性（如尺度、位置和方向）以及独立于场景的表示（即外观和形状）。实验结果证实了所提出方法的有效性，在全球对象中心表示学习、对象识别、特定对象的场景生成及场景分解方面表现出色。
URL
https://arxiv.org/abs/2410.18809
PDF
-
DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes
2024-10-23 17:59:58Hengwei Bian, Lingdong Kong, Haozhe Xie, Liang Pan, Yu Qiao, Ziwei LiuAbstract
LiDAR scene generation has been developing rapidly recently. However, existing methods primarily focus on generating static and single-frame scenes, overlooking the inherently dynamic nature of real-world driving environments. In this work, we introduce DynamicCity, a novel 4D LiDAR generation framework capable of generating large-scale, high-quality LiDAR scenes that capture the temporal evolution of dynamic environments. DynamicCity mainly consists of two key models. 1) A VAE model for learning HexPlane as the compact 4D representation. Instead of using naive averaging operations, DynamicCity employs a novel Projection Module to effectively compress 4D LiDAR features into six 2D feature maps for HexPlane construction, which significantly enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in parallel, which improves both network training efficiency and reconstruction accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded Rollout Operation is proposed to reorganize all six feature planes of the HexPlane as a squared 2D feature map. In particular, various conditions could be introduced in the diffusion or sampling process, supporting versatile 4D generation applications, such as trajectory- and command-driven generation, inpainting, and layout-conditioned generation. Extensive experiments on the CarlaSC and Waymo datasets demonstrate that DynamicCity significantly outperforms existing state-of-the-art 4D LiDAR generation methods across multiple metrics. The code will be released to facilitate future research.
Abstract (translated)
最近，LiDAR场景生成技术得到了快速发展。然而，现有的方法主要集中在生成静态和单帧场景上，忽视了真实驾驶环境中固有的动态特性。在此工作中，我们引入了DynamicCity，这是一种新颖的4D LiDAR生成框架，能够生成大规模、高质量的LiDAR场景，捕捉动态环境的时间演变。DynamicCity主要包括两个关键模型。 1) 一个VAE（变分自编码器）模型用于学习HexPlane作为紧凑的4D表示形式。与使用简单的平均操作不同，DynamicCity采用了一种新颖的投影模块来有效地将4D LiDAR特征压缩成六个2D特征图以构建HexPlane，这显著提升了HexPlane的拟合质量（最多提高12.56 mIoU）。此外，我们还采用了扩展与压缩策略来并行重建3D特征体积，相较于简单地查询每个3D点，这种方法提高了网络训练效率和重构精度（最高可提升7.05 mIoU，加快2.06倍的训练速度，并减少70.84%的内存使用）。 2) 一个基于DiT（Diffusion Transformer）的扩散模型用于生成HexPlane。为了使HexPlane适用于DiT生成，提出了填充滚动操作来重新组织所有六个特征平面为一个方形的2D特征图。特别地，在扩散或采样过程中可以引入各种条件支持多样的4D生成应用，例如轨迹和指令驱动生成、修复（inpainting）以及布局条件下的生成。 在CarlaSC和Waymo数据集上的大量实验表明，DynamicCity在多个指标上显著优于现有的最先进的4D LiDAR生成方法。代码将被公开发布以促进未来的研究。
URL
https://arxiv.org/abs/2410.18084
PDF
-
The Scene Language: Representing Scenes with Programs, Words, and Embeddings
2024-10-22 07:40:20Yunzhi Zhang, Zizhang Li, Matt Zhou, Shangzhe Wu, Jiajun WuAbstract
We introduce the Scene Language, a visual scene representation that concisely and precisely describes the structure, semantics, and identity of visual scenes. It represents a scene with three key components: a program that specifies the hierarchical and relational structure of entities in the scene, words in natural language that summarize the semantic class of each entity, and embeddings that capture the visual identity of each entity. This representation can be inferred from pre-trained language models via a training-free inference technique, given text or image inputs. The resulting scene can be rendered into images using traditional, neural, or hybrid graphics renderers. Together, this forms a robust, automated system for high-quality 3D and 4D scene generation. Compared with existing representations like scene graphs, our proposed Scene Language generates complex scenes with higher fidelity, while explicitly modeling the scene structures to enable precise control and editing.
Abstract (translated)
URL
https://arxiv.org/abs/2410.16770
PDF
-
L3DG: Latent 3D Gaussian Diffusion
2024-10-17 13:19:32Barbara Roessle, Norman M\"uller, Lorenzo Porzi, Samuel Rota Bul\`o, Peter Kontschieder, Angela Dai, Matthias Nie{\ss}nerAbstract
We propose L3DG, the first approach for generative 3D modeling of 3D Gaussians through a latent 3D Gaussian diffusion formulation. This enables effective generative 3D modeling, scaling to generation of entire room-scale scenes which can be very efficiently rendered. To enable effective synthesis of 3D Gaussians, we propose a latent diffusion formulation, operating in a compressed latent space of 3D Gaussians. This compressed latent space is learned by a vector-quantized variational autoencoder (VQ-VAE), for which we employ a sparse convolutional architecture to efficiently operate on room-scale scenes. This way, the complexity of the costly generation process via diffusion is substantially reduced, allowing higher detail on object-level generation, as well as scalability to large scenes. By leveraging the 3D Gaussian representation, the generated scenes can be rendered from arbitrary viewpoints in real-time. We demonstrate that our approach significantly improves visual quality over prior work on unconditional object-level radiance field synthesis and showcase its applicability to room-scale scene generation.
Abstract (translated)
URL
https://arxiv.org/abs/2410.13530
PDF
-
SceneCraft: Layout-Guided 3D Scene Generation
2024-10-11 17:59:58Xiuyu Yang, Yunze Man, Jun-Kun Chen, Yu-Xiong WangAbstract
The creation of complex 3D scenes tailored to user specifications has been a tedious and challenging task with traditional 3D modeling tools. Although some pioneering methods have achieved automatic text-to-3D generation, they are generally limited to small-scale scenes with restricted control over the shape and texture. We introduce SceneCraft, a novel method for generating detailed indoor scenes that adhere to textual descriptions and spatial layout preferences provided by users. Central to our method is a rendering-based technique, which converts 3D semantic layouts into multi-view 2D proxy maps. Furthermore, we design a semantic and depth conditioned diffusion model to generate multi-view images, which are used to learn a neural radiance field (NeRF) as the final scene representation. Without the constraints of panorama image generation, we surpass previous methods in supporting complicated indoor space generation beyond a single room, even as complicated as a whole multi-bedroom apartment with irregular shapes and layouts. Through experimental analysis, we demonstrate that our method significantly outperforms existing approaches in complex indoor scene generation with diverse textures, consistent geometry, and realistic visual quality. Code and more results are available at: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2410.09049
PDF
-
Skyeyes: Ground Roaming using Aerial View Images
2024-09-25 07:21:43Zhiyuan Gao, Wenbin Teng, Gonglin Chen, Jinsen Wu, Ningli Xu, Rongjun Qin, Andrew Feng, Yajie ZhaoAbstract
Integrating aerial imagery-based scene generation into applications like autonomous driving and gaming enhances realism in 3D environments, but challenges remain in creating detailed content for occluded areas and ensuring real-time, consistent rendering. In this paper, we introduce Skyeyes, a novel framework that can generate photorealistic sequences of ground view images using only aerial view inputs, thereby creating a ground roaming experience. More specifically, we combine a 3D representation with a view consistent generation model, which ensures coherence between generated images. This method allows for the creation of geometrically consistent ground view images, even with large view gaps. The images maintain improved spatial-temporal coherence and realism, enhancing scene comprehension and visualization from aerial perspectives. To the best of our knowledge, there are no publicly available datasets that contain pairwise geo-aligned aerial and ground view imagery. Therefore, we build a large, synthetic, and geo-aligned dataset using Unreal Engine. Both qualitative and quantitative analyses on this synthetic dataset display superior results compared to other leading synthesis approaches. See the project page for more results: this https URL.
Abstract (translated)
将基于高空影像的场景生成集成到诸如自动驾驶和游戏等应用中，可以增强三维环境中现实感，但仍然需要创建详细的遮蔽区域内容，并确保实时和一致的渲染。在本文中，我们介绍了Skyeyes，一种新方法，可以使用仅包括高空影像输入来生成逼真的地面视图序列，从而实现地面漫游体验。具体来说，我们结合了3D表示和一致生成视图模型，确保生成的图像之间保持连贯性。这种方法允许在视图缺口较大时创建几何一致的地面视图图像，增强场景理解和可视化。据我们所知，目前没有公开可用的数据集包含成对的高空和地面影像的地理对齐。因此，我们使用Unreal Engine构建了一个大型、合成和地理对齐的数据集。这个合成数据集的质量和数量分析都显示出了比其他顶级合成方法更好的结果。更多结果请查看项目页面：https://this URL。
URL
https://arxiv.org/abs/2409.16685
PDF
-
SpaceBlender: Creating Context-Rich Collaborative Spaces Through Generative 3D Scene Blending
2024-09-20 22:27:31Nels Numan, Shwetha Rajaram, Balasaravanan Thoravi Kumaravel, Nicolai Marquardt, Andrew D. WilsonAbstract
There is increased interest in using generative AI to create 3D spaces for Virtual Reality (VR) applications. However, today's models produce artificial environments, falling short of supporting collaborative tasks that benefit from incorporating the user's physical context. To generate environments that support VR telepresence, we introduce SpaceBlender, a novel pipeline that utilizes generative AI techniques to blend users' physical surroundings into unified virtual spaces. This pipeline transforms user-provided 2D images into context-rich 3D environments through an iterative process consisting of depth estimation, mesh alignment, and diffusion-based space completion guided by geometric priors and adaptive text prompts. In a preliminary within-subjects study, where 20 participants performed a collaborative VR affinity diagramming task in pairs, we compared SpaceBlender with a generic virtual environment and a state-of-the-art scene generation framework, evaluating its ability to create virtual spaces suitable for collaboration. Participants appreciated the enhanced familiarity and context provided by SpaceBlender but also noted complexities in the generative environments that could detract from task focus. Drawing on participant feedback, we propose directions for improving the pipeline and discuss the value and design of blended spaces for different scenarios.
Abstract (translated)
越来越多的人对使用生成式AI创建虚拟现实（VR）应用中的3D空间感兴趣。然而，目前的模型产生的假定环境不足以支持用户物理上下文中的协作任务。为了生成支持VR远程存在的环境，我们引入了SpaceBlender，一种利用生成式AI技术将用户周围物理环境混合到统一虚拟空间的新管道。这个管道通过深度估计、网格对齐和扩散引导的迭代过程，将用户提供的2D图像转换为上下文丰富的3D环境。在初步的自我对照研究中，20名参与者在一对中执行了协同VR偏好图任务，我们将SpaceBlender与通用虚拟环境和最先进的场景生成框架进行了比较，以评估其创建适合协作的虚拟空间的能力。参与者赞赏SpaceBlender提供的增强熟悉度和上下文，但也指出在生成环境中存在复杂性，可能会分散任务关注力。根据参与者的反馈，我们提出了改进管道的方向，并讨论了不同场景下混合空间的值和设计。
URL
https://arxiv.org/abs/2409.13926
PDF
