Abstract
Neural networks trained on datasets such as ImageNet have led to major advances in visual object classification. One obstacle that prevents networks from reasoning more deeply about complex scenes and situations, and from integrating visual knowledge with natural language, like humans do, is their lack of common sense knowledge about the physical world. Videos, unlike still images, contain a wealth of detailed information about the physical world. However, most labelled video datasets represent high-level concepts rather than detailed physical aspects about actions and scenes. In this work, we describe our ongoing collection of the "something-something" database of video prediction tasks whose solutions require a common sense understanding of the depicted situation. The database currently contains more than 100,000 videos across 174 classes, which are defined as caption-templates. We also describe the challenges in crowd-sourcing this data at scale.
Abstract (translated)
在诸如ImageNet等数据集上训练的神经网络导致了视觉对象分类的重大进展。阻碍网络对复杂场景和情况进行更深入推理的障碍，以及像人类一样将视觉知识与自然语言相结合的一个障碍是它们缺乏关于物理世界的常识知识。与静止图像不同，视频包含大量关于物理世界的详细信息。然而，大多数带标签的视频数据集代表了高层概念，而不是关于动作和场景的详细物理方面。在这项工作中，我们描述了我们正在收集的视频预测任务的“某事”数据库，这些数据库的解决方案需要对所描述的情况有一个常识性的理解。该数据库目前包含超过174个类别的100,000多个视频，这些视频被定义为字幕模板。我们还描述了大规模集中这些数据所面临的挑战。
URL
https://arxiv.org/abs/1706.04261
