Abstract
World-model-based imagine-then-act becomes a promising paradigm for robotic manipulation, yet existing approaches typically support either purely image-based forecasting or reasoning over partial 3D geometry, limiting their ability to predict complete 4D scene dynamics. This work proposes a novel embodied 4D world model that enables geometrically consistent, arbitrary-view RGBD generation: given only a single-view RGBD observation as input, the model imagines the remaining viewpoints, which can then be back-projected and fused to assemble a more complete 3D structure across time. To efficiently learn the multi-view, cross-modality generation, we explicitly design cross-view and cross-modality feature fusion that jointly encourage consistency between RGB and depth and enforce geometric alignment across views. Beyond prediction, converting generated futures into actions is often handled by inverse dynamics, which is ill-posed because multiple actions can explain the same transition. We address this with a test-time action optimization strategy that backpropagates through the generative model to infer a trajectory-level latent best matching the predicted future, and a residual inverse dynamics model that turns this trajectory prior into accurate executable actions. Experiments on three datasets demonstrate strong performance on both 4D scene generation and downstream manipulation, and ablations provide practical insights into the key design choices.
Abstract (translated)
基于世界模型的想象然后行动成为机器人操作的一个有前景的新范式，但现有方法通常仅支持纯图像预测或在部分3D几何结构上的推理，这限制了它们预测完整4D场景动态的能力。这项工作提出了一种新型的具身化4D世界模型，该模型能够生成任意视角下的一致RGBD数据：给定单一视角的RGBD观测输入后，模型可以想象其他视点，并将这些视图反投影融合以构建更完整的3D结构（跨时间）。为了高效地学习多视角、跨模态的数据生成，我们设计了跨视图和跨模态特征融合，从而共同促进RGB与深度数据之间的一致性和不同视图间的几何对齐。除了预测之外，通常通过逆动力学将生成的未来转换为动作执行，但由于多种动作可能解释相同的过渡过程，这种做法是不适定的问题。为此，我们提出了一种测试时的动作优化策略，该策略通过生成模型反向传播以推断出与预测未来最匹配的轨迹级潜在变量，并结合残差逆动力学模型将此轨迹先验转化为准确可执行的动作。 实验在三个数据集上进行，展示了在4D场景生成和下游操作任务上的强大性能。消融研究提供了对关键设计选择的实际见解。
URL
https://arxiv.org/abs/2602.09878
