Abstract
3D scene generation has garnered growing attention in recent years and has made significant progress. Generating 4D cities is more challenging than 3D scenes due to the presence of structurally complex, visually diverse objects like buildings and vehicles, and heightened human sensitivity to distortions in urban environments. To tackle these issues, we propose CityDreamer4D, a compositional generative model specifically tailored for generating unbounded 4D cities. Our main insights are 1) 4D city generation should separate dynamic objects (e.g., vehicles) from static scenes (e.g., buildings and roads), and 2) all objects in the 4D scene should be composed of different types of neural fields for buildings, vehicles, and background stuff. Specifically, we propose Traffic Scenario Generator and Unbounded Layout Generator to produce dynamic traffic scenarios and static city layouts using a highly compact BEV representation. Objects in 4D cities are generated by combining stuff-oriented and instance-oriented neural fields for background stuff, buildings, and vehicles. To suit the distinct characteristics of background stuff and instances, the neural fields employ customized generative hash grids and periodic positional embeddings as scene parameterizations. Furthermore, we offer a comprehensive suite of datasets for city generation, including OSM, GoogleEarth, and CityTopia. The OSM dataset provides a variety of real-world city layouts, while the Google Earth and CityTopia datasets deliver large-scale, high-quality city imagery complete with 3D instance annotations. Leveraging its compositional design, CityDreamer4D supports a range of downstream applications, such as instance editing, city stylization, and urban simulation, while delivering state-of-the-art performance in generating realistic 4D cities.
Abstract (translated)
近年来，三维场景生成受到了越来越多的关注，并取得了显著进展。然而，生成四维城市（包含时间维度的变化）比单纯生成三维场景更具挑战性，这是因为建筑物和车辆等结构复杂、视觉多样化的物体的存在以及人们对城市环境中任何失真的高度敏感性。为了解决这些问题，我们提出了CityDreamer4D，这是一种专门用于生成无边界四维城市的组合式生成模型。 我们的主要见解是： 1. 四维城市的生成应该将动态对象（例如车辆）与静态场景（例如建筑物和道路）区分开来。 2. 在四维场景中的所有物体都应由不同类型神经场组成，包括针对建筑物、车辆及背景物质的神经场。 具体来说，我们提出了交通场景生成器和无边界布局生成器，这两个生成器使用高度紧凑的BEV（鸟瞰图）表示方式来产生动态交通场景和静态城市布局。在四维城市中，物体通过结合面向物质和实例化的神经场（背景物质、建筑物和车辆）进行生成。 为了适应背景物质和实例的不同特性，这些神经场采用了定制化生成哈希网格和周期位置嵌入作为场景参数化方式。 此外，我们提供了一整套用于城市生成的数据集，包括OSM、Google Earth 和 CityTopia。其中OSM数据集提供了多种真实世界的城市布局，而 Google 地球和CityTopia 数据集则提供了大规模、高质量的城市图像，并附有3D实例标注信息。借助其组合式设计，CityDreamer4D 支持一系列下游应用，包括实例编辑、城市风格化及都市仿真等，同时在生成逼真的四维城市方面表现出色。
URL
https://arxiv.org/abs/2501.08983
