Abstract
Temporal grounding is the task of locating a specific segment from an untrimmed video according to a query sentence. This task has achieved significant momentum in the computer vision community as it enables activity grounding beyond pre-defined activity classes by utilizing the semantic diversity of natural language descriptions. The semantic diversity is rooted in the principle of compositionality in linguistics, where novel semantics can be systematically described by combining known words in novel ways (compositional generalization). However, existing temporal grounding datasets are not carefully designed to evaluate the compositional generalizability. To systematically benchmark the compositional generalizability of temporal grounding models, we introduce a new Compositional Temporal Grounding task and construct two new dataset splits, i.e., Charades-CG and ActivityNet-CG. When evaluating the state-of-the-art methods on our new dataset splits, we empirically find that they fail to generalize to queries with novel combinations of seen words. We argue that the inherent structured semantics inside the videos and language is the crucial factor to achieve compositional generalization. Based on this insight, we propose a variational cross-graph reasoning framework that explicitly decomposes video and language into hierarchical semantic graphs, respectively, and learns fine-grained semantic correspondence between the two graphs. Furthermore, we introduce a novel adaptive structured semantics learning approach to derive the structure-informed and domain-generalizable graph representations, which facilitate the fine-grained semantic correspondence reasoning between the two graphs. Extensive experiments validate the superior compositional generalizability of our approach.
Abstract (translated)
时间基座任务是确定从未剪辑的视频中提取的特定片段,根据查询语句的位置。这个任务在计算机视觉社区中取得了显著的进展,因为它可以利用自然语言描述语义的多样性,实现超出预先定义的活动类别的活动基座。语义多样性的根源在于语言学中的组合性原则,在那里,新的语义可以用新的方式组合已知的单词 systematically 描述(组合扩展)。然而,现有的时间基座数据集没有精心设计以评估组合扩展的通用性。为了系统性地基准时间基座模型的时间组合扩展通用性,我们介绍了一个新的组合时间基座任务,并构建了两个新的数据集分裂,即Charades-CG和ActivityNet-CG。当我们评估我们新数据集分裂中的方法时,我们Empirically 发现它们无法泛化到看到单词的新颖组合。我们指出,视频和语言的内部结构性语义是实现组合扩展的关键因素。基于这个见解,我们提出了一种异构的交叉 graph 推理框架,将视频和语言明确分解为Hierarchical 语义 Graph,并学习两个 Graph 之间的精细语义对应。此外,我们介绍了一种新颖的自适应结构化语义学习方法,以推导结构 informed 和域 generalizable 的Graph 表示,这促进了两个 Graph 之间的精细语义对应推理。广泛的实验验证了我们方法的高度组合扩展通用性。
URL
https://arxiv.org/abs/2301.09071
