Abstract
Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements. We propose an approach to learn human-object interaction "hotspots" directly from video. Rather than treat affordances as a manually supervised semantic segmentation task, our approach learns about interactions by watching videos of real human behavior and anticipating afforded actions. Given a novel image or video, our model infers a spatial hotspot map indicating how an object would be manipulated in a potential interaction, even if the object is currently at rest. Through results with both first and third person video, we show the value of grounding affordances in real human-object interactions. Not only are our weakly supervised hotspots competitive with strongly supervised affordance methods, but they can also anticipate object interaction for novel object categories. Project page: this http URL
Abstract (translated)
学习如何与物体交互是实现视觉智能的重要一步，但现有的技术面临着繁重的监控或传感要求。我们提出了一种直接从视频中学习人-物交互“热点”的方法。我们的方法不是将提供视为一项人工监督的语义分割任务，而是通过观看真实人类行为的视频和预测提供的行为来学习交互。给出一个新的图像或视频，我们的模型推断出一个空间热点图，表明一个对象如何在潜在的交互中被操纵，即使该对象当前处于静止状态。通过第一人称和第三人称视频的结果，我们展示了在真实的人-物交互中接地供给的价值。我们的弱监督热点不仅与强监督供给方法竞争，而且还可以预测新对象类别的对象交互。项目页：此HTTP URL
URL
https://arxiv.org/abs/1906.01963
