Abstract
Video repetition counting infers the number of repetitions of recurring actions or motion within a video. We propose an exemplar-based approach that discovers visual correspondence of video exemplars across repetitions within target videos. Our proposed Every Shot Counts (ESCounts) model is an attention-based encoder-decoder that encodes videos of varying lengths alongside exemplars from the same and different videos. In training, ESCounts regresses locations of high correspondence to the exemplars within the video. In tandem, our method learns a latent that encodes representations of general repetitive motions, which we use for exemplar-free, zero-shot inference. Extensive experiments over commonly used datasets (RepCount, Countix, and UCFRep) showcase ESCounts obtaining state-of-the-art performance across all three datasets. On RepCount, ESCounts increases the off-by-one from 0.39 to 0.56 and decreases the mean absolute error from 0.38 to 0.21. Detailed ablations further demonstrate the effectiveness of our method.
Abstract (translated)
视频重复计数推断了视频内重复动作或运动的次数。我们提出了一种基于示例的 approach，发现了目标视频中重复示例之间的视觉对应关系。我们提出的 Every Shot Counts (ESCounts) 模型是一种自注意力编码器-解码器，它与来自相同和不同视频的示例一起编码不同长度的视频。在训练过程中，ESCounts 回归视频内高相似度的示例位置。与此同时，我们的方法学习了一个隐含表示，编码了通用重复运动的表示。我们使用这个隐含表示进行无示例、零示例推理。在常用数据集（RepCount、Countix 和 UCFRep）上进行的大量实验展示了 ESCounts 在三个数据集上的最佳性能。在 RepCount 上，ESCounts 将 off-by-one 从 0.39 增加到了 0.56，将平均绝对误差从 0.38 降低到了 0.21。进一步的详细分析证明了我们的方法的的有效性。
URL
https://arxiv.org/abs/2403.18074
