Abstract
Temporal consistency is critical in video prediction to ensure that outputs are coherent and free of artifacts. Traditional methods, such as temporal attention and 3D convolution, may struggle with significant object motion and may not capture long-range temporal dependencies in dynamic scenes. To address this gap, we propose the Tracktention Layer, a novel architectural component that explicitly integrates motion information using point tracks, i.e., sequences of corresponding points across frames. By incorporating these motion cues, the Tracktention Layer enhances temporal alignment and effectively handles complex object motions, maintaining consistent feature representations over time. Our approach is computationally efficient and can be seamlessly integrated into existing models, such as Vision Transformers, with minimal modification. It can be used to upgrade image-only models to state-of-the-art video ones, sometimes outperforming models natively designed for video prediction. We demonstrate this on video depth prediction and video colorization, where models augmented with the Tracktention Layer exhibit significantly improved temporal consistency compared to baselines.
Abstract (translated)
时间一致性在视频预测中至关重要，以确保输出的连贯性和无瑕疵。传统方法，如时序注意力和3D卷积，在处理显著的对象运动以及捕捉动态场景中的长程时序依赖关系方面可能会遇到困难。为了解决这一缺口，我们提出了Tracktention层，这是一个新颖的架构组件，它通过点轨迹（即帧间对应的点序列）显式地整合了运动信息。通过引入这些运动线索，Tracktention 层增强了时序对齐，并有效地处理复杂的对象运动，在整个时间段内保持一致的功能表示。我们的方法计算效率高，可以无缝集成到现有的模型中，如视觉变换器（Vision Transformers），只需进行少量修改即可。它可以用于将仅基于图像的模型升级为最先进的视频模型，在某些情况下甚至超过了原生设计用于视频预测的模型性能。我们在视频深度预测和视频上色任务中展示了这一点，其中增强有 Tracktention 层的模型在时间一致性方面显著优于基线模型。
URL
https://arxiv.org/abs/2503.19904
