Abstract
We aim to perform unsupervised discovery of objects and their states such as location and velocity, as well as physical system parameters such as mass and gravity from video -- given only the differential equations governing the scene dynamics. Existing physical scene understanding methods require either object state supervision, or do not integrate with differentiable physics to learn interpretable system parameters and states. We address this problem through a $\textit{physics-as-inverse-graphics}$ approach that brings together vision-as-inverse-graphics and differentiable physics engines. This framework allows us to perform long term extrapolative video prediction, as well as vision-based model-predictive control. Our approach significantly outperforms related unsupervised methods in long-term future frame prediction of systems with interacting objects (such as ball-spring or 3-body gravitational systems). We further show the value of this tight vision-physics integration by demonstrating data-efficient learning of vision-actuated model-based control for a pendulum system. The controller's interpretability also provides unique capabilities in goal-driven control and physical reasoning for zero-data adaptation.
Abstract (translated)
我们的目标是对物体及其状态（如位置和速度）以及物理系统参数（如视频中的质量和重力）进行无监督的发现——只给出控制场景动力学的微分方程。现有的物理场景理解方法要么需要对象状态监控，要么不与可微物理结合来学习可解释的系统参数和状态。我们通过一个$ extit物理作为逆图形方法来解决这个问题，该方法将视觉作为逆图形和可微物理引擎结合在一起。这个框架允许我们进行长期的外推视频预测，以及基于视觉的模型预测控制。我们的方法在对具有相互作用物体（如球弹簧或3体引力系统）的系统进行长期帧预测时，明显优于相关的无监督方法。通过对摆锤系统视觉驱动模型控制的数据有效学习，进一步证明了这种紧密视觉物理集成的价值。控制器的可解释性还提供了目标驱动控制和零数据自适应物理推理方面的独特功能。
URL
https://arxiv.org/abs/1905.11169
