Abstract
Predicting future motion is crucial in video understanding and controllable video generation. Dense point trajectories are a compact, expressive motion representation, but modeling their future evolution from observed video remains challenging. We propose a framework that predicts future trajectories and visibility from past trajectories and video context. Our method has three components: (1) Grid-Anchor Offset Encoding, which reduces location-dependent bias by representing each point as an offset from its pixel-center anchor; (2) TrajLoom-VAE, which learns a compact spatiotemporal latent space for dense trajectories with masked reconstruction and a spatiotemporal consistency regularizer; and (3) TrajLoom-Flow, which generates future trajectories in latent space via flow matching, with boundary cues and on-policy K-step fine-tuning for stable sampling. We also introduce TrajLoomBench, a unified benchmark spanning real and synthetic videos with a standardized setup aligned with video-generation benchmarks. Compared with state-of-the-art methods, our approach extends the prediction horizon from 24 to 81 frames while improving motion realism and stability across datasets. The predicted trajectories directly support downstream video generation and editing. Code, model checkpoints, and datasets are available at this https URL.
Abstract (translated)
预测未来运动在视频理解与可控视频生成中至关重要。密集点轨迹是一种紧凑且富有表现力的运动表征，但从观测视频中建模其未来演变仍具有挑战性。我们提出了一种从历史轨迹与视频上下文预测未来轨迹及可见性的框架。该方法包含三个核心组件：(1) 网格锚点偏移编码（Grid-Anchor Offset Encoding），通过将每个点表示为其像素中心锚点的偏移量，减轻位置依赖偏差；(2) TrajLoom-VAE，借助掩码重建与时空一致性正则化器，为密集轨迹学习紧凑的时空潜在空间；(3) TrajLoom-Flow，通过流匹配在潜在空间中生成未来轨迹，并利用边界线索与同策略K步微调实现稳定采样。我们还引入了TrajLoomBench，这是一个覆盖真实与合成视频的统一基准，其标准化设置与视频生成基准对齐。相较于最先进方法，我们的方案将预测范围从24帧扩展到81帧，同时在跨数据集上提升了运动真实性与稳定性。预测轨迹可直接支持下游视频生成与编辑任务。代码、模型检查点及数据集已公开于此链接。
URL
https://arxiv.org/abs/2603.22606
