Abstract
Extracting optical flow from videos remains a core computer vision problem. Motivated by the success of large general-purpose models, we ask whether frozen self-supervised video models trained only for future frame prediction can be prompted, without fine-tuning, to output flow. Prior work reading out depth or illumination from video generators required fine-tuning, which is impractical for flow where labels are scarce and synthetic datasets suffer from a sim-to-real gap. Inspired by the Counterfactual World Model (CWM) paradigm, which can obtain point-wise correspondences by injecting a small tracer perturbation into a next-frame predictor and tracking its propagation, we extend this idea to generative video models. We explore several popular architectures and find that successful zero-shot flow extraction in this manner is aided by three model properties: (1) distributional prediction of future frames (avoiding blurry or noisy outputs); (2) factorized latents that treat each spatio-temporal patch independently; and (3) random-access decoding that can condition on any subset of future pixels. These properties are uniquely present in the recent Local Random Access Sequence (LRAS) architecture. Building on LRAS, we propose KL-tracing: a novel test-time procedure that injects a localized perturbation into the first frame, rolls out the model one step, and computes the Kullback-Leibler divergence between perturbed and unperturbed predictive distributions. Without any flow-specific fine-tuning, our method outperforms state-of-the-art models on real-world TAP-Vid DAVIS dataset (16.6% relative improvement for endpoint error) and synthetic TAP-Vid Kubric (4.7% relative improvement). Our results indicate that counterfactual prompting of controllable generative video models is a scalable and effective alternative to supervised or photometric-loss approaches for high-quality flow.
Abstract (translated)
从视频中提取光流仍然是计算机视觉领域的一个核心问题。鉴于大型通用模型的成功，我们探讨了冻结的仅用于未来帧预测的自监督视频模型是否可以在不进行微调的情况下被提示以输出光流。先前的工作通过读取视频生成器中的深度或光照信息时通常需要进行微调，这在光流提取中是不可行的，因为光流标签稀缺且合成数据集存在仿真到现实的差距问题。 受反事实世界模型（Counterfactual World Model, CWM）范式的启发，该方法可以通过向下一帧预测器注入一个小的扰动追踪器并跟踪其传播来获得点对点对应关系，我们将这个想法扩展到了生成式视频模型上。我们探索了几种流行的架构，并发现以这种方式成功实现零样本光流提取需满足三个模型特性：(1) 对未来帧进行分布预测（避免模糊或噪声输出）；(2) 因子化潜在变量，将每个时空补丁独立处理；以及 (3) 随机访问解码器可以基于任何未来的像素子集进行条件化。这些特性在最近的局部随机访问序列（Local Random Access Sequence, LRAS）架构中独一无二地存在。 基于LRAS架构，我们提出了一种新的测试时过程：KL-tracing。该方法向第一帧注入一个局域扰动，模型滚动一步后计算扰动与未扰动预测分布之间的Kullback-Leibler散度。无需任何特定于光流的微调，我们的方法在真实世界的TAP-Vid DAVIS数据集上超越了当前最优的方法（以端点误差衡量，相对改进16.6%），并在合成的TAP-Vid Kubric数据集上实现了4.7%的相对改善。 研究结果表明，对可控生成式视频模型进行反事实提示是一种可扩展且有效的替代方法，它能为高质量光流提供比监督学习或光度损失方法更好的解决方案。
URL
https://arxiv.org/abs/2507.09082
