Abstract
Video is a promising source of knowledge for embodied agents to learn models of the world's dynamics. Large deep networks have become increasingly effective at modeling complex video data in a self-supervised manner, as evaluated by metrics based on human perceptual similarity or pixel-wise comparison. However, it remains unclear whether current metrics are accurate indicators of performance on downstream tasks. We find empirically that for planning robotic manipulation, existing metrics can be unreliable at predicting execution success. To address this, we propose a benchmark for action-conditioned video prediction in the form of a control benchmark that evaluates a given model for simulated robotic manipulation through sampling-based planning. Our benchmark, Video Prediction for Visual Planning ($VP^2$), includes simulated environments with 11 task categories and 310 task instance definitions, a full planning implementation, and training datasets containing scripted interaction trajectories for each task category. A central design goal of our benchmark is to expose a simple interface -- a single forward prediction call -- so it is straightforward to evaluate almost any action-conditioned video prediction model. We then leverage our benchmark to study the effects of scaling model size, quantity of training data, and model ensembling by analyzing five highly-performant video prediction models, finding that while scale can improve perceptual quality when modeling visually diverse settings, other attributes such as uncertainty awareness can also aid planning performance.
Abstract (translated)
视频是身体参与 agents 学习世界动态模型的有前途的知识来源。大型深网络在以自我监督方式建模复杂的视频数据方面变得越来越有效，以基于人类感知相似性或像素比较的度量指标进行评估。然而，目前仍不清楚当前度量指标是否准确地反映了下游任务的表现。我们经验证，对于规划机器人操纵，现有的度量指标在预测执行成功方面的可靠性是不可靠的。为了解决这个问题，我们提出了一种控制基准，以作为评估通过采样计划模拟机器人操纵给定模型的基准。我们的基准是“视频预测Visual Planning”(VP2)，包括模拟环境，每个任务类别具有11个任务分类和310个任务实例定义，以及完整的计划实施和包含每个任务类别的编程交互轨迹的训练数据集。我们的基准的核心设计目标是暴露一个简单的接口——一个单一的预测向前调用——以便几乎可以直接评估任何行动条件的视频预测模型。然后我们利用我们的基准来分析五家表现优异的视频预测模型，发现虽然规模可以在建模视觉多样性的情境中改善感知质量，但其他属性，如不确定性意识，也可以协助规划性能。
URL
https://arxiv.org/abs/2304.13723
