Abstract
Temporal video alignment aims to synchronize the key events like object interactions or action phase transitions in two videos. Such methods could benefit various video editing, processing, and understanding tasks. However, existing approaches operate under the restrictive assumption that a suitable video pair for alignment is given, significantly limiting their broader applicability. To address this, we re-pose temporal alignment as a search problem and introduce the task of Alignable Video Retrieval (AVR). Given a query video, our approach can identify well-alignable videos from a large collection of clips and temporally synchronize them to the query. To achieve this, we make three key contributions: 1) we introduce DRAQ, a video alignability indicator to identify and re-rank the best alignable video from a set of candidates; 2) we propose an effective and generalizable frame-level video feature design to improve the alignment performance of several off-the-shelf feature representations, and 3) we propose a novel benchmark and evaluation protocol for AVR using cycle-consistency metrics. Our experiments on 3 datasets, including large-scale Kinetics700, demonstrate the effectiveness of our approach in identifying alignable video pairs from diverse datasets. Project Page: this https URL.
Abstract (translated)
temporal video alignment 旨在将两个视频中的关键事件（如物体交互或动作阶段转换）同步。这类方法可以为各种视频编辑、处理和理解任务提供有益的帮助。然而，现有方法在假设给出合适的视频对进行对齐时非常有限，这使得它们的适用性受到很大限制。为了解决这个问题，我们将 temporal alignment 重新定位为搜索问题，并引入了 Alignable Video Retrieval（AVR）任务。给定查询视频，我们的方法可以从大量视频片段中找到 well-alignable 视频，并将其与查询视频同步。为实现这一目标，我们做出了以下三个关键贡献：1）我们引入了 DRAQ，这是一种视频对齐度指标，用于识别和重新排名一组候选视频中的最佳对齐视频；2）我们提出了一个有效且通用视频特征设计，以提高多个离散特征表示的同步性能；3）我们提出了使用循环一致性度量的新 benchmark 和评估协议 for AVR。我们对 3 个数据集的实验结果表明，我们的方法在从多样数据集中确定对齐视频对方面非常有效。项目页面：此链接。
URL
https://arxiv.org/abs/2409.01445
