Abstract
Current video generation models cannot simulate physical consequences of 3D actions like forces and robotic manipulations, as they lack structural understanding of how actions affect 3D scenes. We present RealWonder, the first real-time system for action-conditioned video generation from a single image. Our key insight is using physics simulation as an intermediate bridge: instead of directly encoding continuous actions, we translate them through physics simulation into visual representations (optical flow and RGB) that video models can process. RealWonder integrates three components: 3D reconstruction from single images, physics simulation, and a distilled video generator requiring only 4 diffusion steps. Our system achieves 13.2 FPS at 480x832 resolution, enabling interactive exploration of forces, robot actions, and camera controls on rigid objects, deformable bodies, fluids, and granular materials. We envision RealWonder opens new opportunities to apply video models in immersive experiences, AR/VR, and robot learning. Our code and model weights are publicly available in our project website: this https URL
Abstract (translated)
当前的视频生成模型无法模拟3D动作（如力和机器人操作）对物理世界的连续影响，因为它们缺乏对动作如何影响3D场景的结构理解。我们提出了一种名为RealWonder的新系统，这是首个基于单张图像进行动作条件下的实时视频生成系统的解决方案。 我们的关键洞察在于使用物理学仿真作为中间桥梁：不是直接编码连续的动作，而是通过物理学仿真将其转换为视觉表示（光流和RGB），这些是视频模型能够处理的。 RealWonder集成三大组件：从单一图片重建3D场景、物理仿真以及一个仅需4次扩散步骤的精简视频生成器。 该系统在480x832分辨率下可达到13.2帧每秒的速度，支持刚体对象、可变形物体、流体和颗粒材料上的力、机器人动作及相机控制交互式的探索。我们设想RealWonder将为视频模型在沉浸式体验、AR/VR以及机器人学习中的应用开辟新的机会。 我们的代码和模型权重可在项目网站上公开获取：[此URL](this https URL)
URL
https://arxiv.org/abs/2603.05449
