Abstract
World models have garnered increasing attention in the development of artificial general intelligence (AGI), serving as computational frameworks for learning representations of the external world and forecasting future states. While early efforts focused on 2D visual perception and simulation, recent 3D-aware generative world models have demonstrated the ability to synthesize geometrically consistent, interactive 3D environments, marking a shift toward 3D spatial cognition. Despite rapid progress, the field lacks systematic analysis to categorize emerging techniques and clarify their roles in advancing 3D cognitive world models. This survey addresses this need by introducing a conceptual framework, providing a structured and forward-looking review of world models transitioning from 2D perception to 3D cognition. Within this framework, we highlight two key technological drivers, particularly advances in 3D representations and the incorporation of world knowledge, as fundamental pillars. Building on these, we dissect three core cognitive capabilities that underpin 3D world modeling: 3D physical scene generation, 3D spatial reasoning, and 3D spatial interaction. We further examine the deployment of these capabilities in real-world applications, including embodied AI, autonomous driving, digital twin, and gaming/VR. Finally, we identify challenges across data, modeling, and deployment, and outline future directions for advancing more robust and generalizable 3D world models.
Abstract (translated)
世界模型在人工通用智能（AGI）的发展中获得了越来越多的关注，它们作为计算框架用于学习外部世界的表示并预测未来的状态。早期的努力主要集中在二维视觉感知和模拟上，而最近基于三维感知的生成型世界模型则展示了合成几何一致且可互动的3D环境的能力，标志着向三维空间认知的重大转变。尽管这一领域取得了迅速的进步，但缺乏系统性的分析来分类新兴技术，并明确它们在推进三维认知世界模型中的作用。本综述旨在通过引入概念框架，提供一个结构化和前瞻性的关于从二维感知到三维认知的世界模型的审查，以满足这一需求。 在这个框架内，我们强调了两项关键的技术驱动力：三维表示的进步和世界知识的整合，作为基础支柱。在此基础上，我们将剖析三个核心的认知能力，这些能力支撑着3D世界的建模：3D物理场景生成、3D空间推理以及3D空间互动。此外，我们将进一步探讨在现实应用中部署这些能力的情况，包括具身人工智能、自动驾驶汽车、数字孪生及游戏/虚拟现实等。 最后，我们识别了跨数据、模型和部署层面的挑战，并概述了推进更强大且通用化的三维世界模型的未来方向。
URL
https://arxiv.org/abs/2506.20134
