Abstract
Forecasting the evolution of 3D scenes and generating unseen scenarios via occupancy-based world models offers substantial potential for addressing corner cases in autonomous driving systems. While tokenization has revolutionized image and video generation, efficiently tokenizing complex 3D scenes remains a critical challenge for 3D world models. To address this, we propose $I^{2}$-World, an efficient framework for 4D occupancy forecasting. Our method decouples scene tokenization into intra-scene and inter-scene tokenizers. The intra-scene tokenizer employs a multi-scale residual quantization strategy to hierarchically compress 3D scenes while preserving spatial details. The inter-scene tokenizer residually aggregates temporal dependencies across timesteps. This dual design preserves the compactness of 3D tokenizers while retaining the dynamic expressiveness of 4D tokenizers. Unlike decoder-only GPT-style autoregressive models, $I^{2}$-World adopts an encoder-decoder architecture. The encoder aggregates spatial context from the current scene and predicts a transformation matrix to enable high-level control over scene generation. The decoder, conditioned on this matrix and historical tokens, ensures temporal consistency during generation. Experiments demonstrate that $I^{2}$-World achieves state-of-the-art performance, outperforming existing methods by 25.1\% in mIoU and 36.9\% in IoU for 4D occupancy forecasting while exhibiting exceptional computational efficiency: it requires merely 2.9 GB of training memory and achieves real-time inference at 37.0 FPS. Our code is available on this https URL.
Abstract (translated)
基于占用的模型预测三维场景的发展并生成未见过的情景，为解决自动驾驶系统中的边缘情况提供了巨大的潜力。虽然标记化技术在图像和视频生成领域带来了革命性的变革，但高效地对复杂的三维场景进行标记仍然是三维世界模型面临的一个关键挑战。为此，我们提出了$I^{2}$-World，这是一个用于四维占用预测的高效框架。我们的方法将场景标记分解为内部场景标记器（intra-scene tokenizer）和跨场景标记器（inter-scene tokenizer）。 **内部场景标记器**采用多尺度残差量化策略来分层压缩三维场景，同时保留空间细节；而**跨场景标记器**则通过逐时间步的方式聚合时序依赖性。这种双管齐下的设计在保持3D标记器的紧凑性的同时，还保留了4D标记器的动态表达能力。 与解码器独占的GPT风格的自回归模型不同，$I^{2}$-World采用了一种编码器-解码器架构。编码器从当前场景中聚合空间上下文，并预测一个转换矩阵以实现对场景生成的高级控制；而解码器则根据该矩阵和历史标记，在生成过程中确保时间一致性。 实验表明，$I^{2}$-World在四维占用预测方面取得了最先进的性能，与现有方法相比，其mIoU（平均交并比）提高了25.1%，IoU（交并比）提高了36.9%。此外，它还展示了卓越的计算效率：仅需2.9 GB的训练内存，并在实时推理中达到每秒37.0帧的速度。 我们的代码可以在以下链接获取: [请在此处插入URL]。
URL
https://arxiv.org/abs/2507.09144
