Abstract
Forecasting from partial observations is central to world modeling. Many recent methods represent the world through images, and reduce forecasting to stochastic video generation. Although such methods excel at realism and visual fidelity, predicting pixels is computationally intensive and not directly useful in many applications, as it requires translating RGB into signals useful for decision making. An alternative approach uses features from vision foundation models (VFMs) as world representations, performing deterministic regression to predict future world states. These features can be directly translated into actionable signals such as semantic segmentation and depth, while remaining computationally efficient. However, deterministic regression averages over multiple plausible futures, undermining forecast accuracy by failing to capture uncertainty. To address this crucial limitation, we introduce a generative forecaster that performs autoregressive flow matching in VFM feature space. Our key insight is that generative modeling in this space requires encoding VFM features into a compact latent space suitable for diffusion. We show that this latent space preserves information more effectively than previously used PCA-based alternatives, both for forecasting and other applications, such as image generation. Our latent predictions can be easily decoded into multiple useful and interpretable output modalities: semantic segmentation, depth, surface normals, and even RGB. With matched architecture and compute, our method produces sharper and more accurate predictions than regression across all modalities. Our results suggest that stochastic conditional generation of VFM features offers a promising and scalable foundation for future world models.
Abstract (translated)
从部分观测数据进行预测是世界建模的核心。许多近期的方法通过图像来表示世界，并将预测简化为随机视频生成任务。尽管这些方法在现实感和视觉保真度方面表现出色，但像素级的预测计算成本高且在很多应用场景中不够直接有用，因为它们需要将RGB信息转换成可用于决策制定的有效信号。另一种方法则是利用计算机视觉基础模型（VFMs）中的特征作为世界表示方式，并执行确定性回归以预测未来的状态。这些特征可以直接转化为语义分割、深度等行动信号，同时保持计算效率。然而，这种确定性的回归方法通过平均多个可能的未来结果来削弱了预测的准确性，因为它无法捕捉不确定性。 为了解决这个关键限制，我们引入了一种生成式预报器，在VFMs特性空间中执行自回归流匹配（autoregressive flow matching）。我们的核心见解在于：在这种空间中进行生成建模需要将VFMs特征编码到适合扩散过程的紧凑隐变量空间。研究表明这种隐变量空间比以前使用的基于PCA的方法更有效地保存信息，不仅在预测方面，在图像生成等其他应用上也是如此。我们模型产生的潜在结果可以轻松解码为多种有用且可解释的输出模式：语义分割、深度、表面法线甚至RGB。 与回归方法相比，使用相同架构和计算资源的情况下，我们的方法能够对所有模态产生更清晰准确的预测。研究表明，基于VFMs特征进行条件生成的方法可能为未来世界模型提供了有前途且具有可扩展性的基础。
URL
https://arxiv.org/abs/2512.11225
