Abstract
Our objective is to discover and localize monotonic temporal changes in a sequence of images. To achieve this, we exploit a simple proxy task of ordering a shuffled image sequence, with `time' serving as a supervisory signal since only changes that are monotonic with time can give rise to the correct ordering. We also introduce a flexible transformer-based model for general-purpose ordering of image sequences of arbitrary length with built-in attribution maps. After training, the model successfully discovers and localizes monotonic changes while ignoring cyclic and stochastic ones. We demonstrate applications of the model in multiple video settings covering different scene and object types, discovering both object-level and environmental changes in unseen sequences. We also demonstrate that the attention-based attribution maps function as effective prompts for segmenting the changing regions, and that the learned representations can be used for downstream applications. Finally, we show that the model achieves the state of the art on standard benchmarks for ordering a set of images.
Abstract (translated)
我们的目标是发现和局部化图像序列中的单调时间变化。为了实现这一目标，我们利用了一个简单的代理任务，即对随机图像序列进行排序，其中`time'作为监督信号，因为只有与时间相关的单调变化才能得到正确的排序。我们还引入了一个灵活的Transformer-based模型，用于对任意长度的图像序列进行通用排序，并内置归一化映射。在训练之后，该模型在成功发现和局部化单调变化的同时，忽略了循环和随机变化。我们在多个视频设置中展示了该模型的应用，涵盖了不同的场景和对象类型，发现了未见过的序列中的物体级和环境变化。我们还证明了基于注意的归一化映射可以作为分割变化区域的有效提示，并且学到的表示可以用于下游应用。最后，我们证明了该模型在为给定一组图像排序的基准测试中达到了最先进的水平。
URL
https://arxiv.org/abs/2404.16828
