Abstract
Mesh models have become increasingly accessible for numerous cities; however, the lack of realistic textures restricts their application in virtual urban navigation and autonomous driving. To address this, this paper proposes MeSS (Meshbased Scene Synthesis) for generating high-quality, styleconsistent outdoor scenes with city mesh models serving as the geometric prior. While image and video diffusion models can leverage spatial layouts (such as depth maps or HD maps) as control conditions to generate street-level perspective views, they are not directly applicable to 3D scene generation. Video diffusion models excel at synthesizing consistent view sequences that depict scenes but often struggle to adhere to predefined camera paths or align accurately with rendered control videos. In contrast, image diffusion models, though unable to guarantee cross-view visual consistency, can produce more geometry-aligned results when combined with ControlNet. Building on this insight, our approach enhances image diffusion models by improving cross-view consistency. The pipeline comprises three key stages: first, we generate geometrically consistent sparse views using Cascaded Outpainting ControlNets; second, we propagate denser intermediate views via a component dubbed AGInpaint; and third, we globally eliminate visual inconsistencies (e.g., varying exposure) using the GCAlign module. Concurrently with generation, a 3D Gaussian Splatting (3DGS) scene is reconstructed by initializing Gaussian balls on the mesh surface. Our method outperforms existing approaches in both geometric alignment and generation quality. Once synthesized, the scene can be rendered in diverse styles through relighting and style transfer techniques.
Abstract (translated)
网格模型（Mesh models）对于许多城市来说已经变得越来越容易获取；然而，缺乏逼真的纹理限制了它们在虚拟城市导航和自动驾驶中的应用。为了应对这一问题，本文提出了 MeSS (基于网格的场景合成)，旨在利用城市的网格模型作为几何先验条件来生成高质量、风格一致的户外场景。虽然图像和视频扩散模型可以借助空间布局（如深度图或高清地图）作为控制条件来生成街道视角视图，但它们并不直接适用于3D场景生成。视频扩散模型擅长于合成描绘场景的一致性视图序列，但在遵循预定义的相机路径或与渲染的控制视频对齐方面常常遇到困难。相比之下，尽管图像扩散模型无法保证跨视角视觉一致性，但如果结合使用ControlNet，则可以产生更具几何一致性的结果。基于这一洞察，我们的方法通过改进图像扩散模型之间的视角一致性来增强它们的功能。 该流程包含三个关键阶段：首先，我们利用级联Outpainting ControlNets生成几何上一致的稀疏视图；其次，我们将更密集的中间视图通过名为AGInpaint的组件进行传播；第三，我们使用GCAlign模块全局消除视觉不一致性（例如不同的曝光度）。与此同时，在生成过程中，通过在网格表面上初始化高斯球来重建3D高斯点阵（3DGS）场景。我们的方法在几何对齐和生成质量方面都优于现有的方法。一旦合成完成，可以通过重新照明和风格转移技术以多种样式渲染场景。 这种综合方法不仅提升了基于图像扩散模型的三维场景生成能力，还为未来在虚拟城市导航和自动驾驶领域中的应用铺平了道路，使得利用现有网格数据创建逼真的虚拟环境成为可能。
URL
https://arxiv.org/abs/2508.15169
