Abstract
Reality is a dance between rigid constraints and deformable structures. For video models, that means generating motion that preserves fidelity as well as structure. Despite progress in diffusion models, producing realistic structure-preserving motion remains challenging, especially for articulated and deformable objects such as humans and animals. Scaling training data alone, so far, has failed to resolve physically implausible transitions. Existing approaches rely on conditioning with noisy motion representations, such as optical flow or skeletons extracted using an external imperfect model. To address these challenges, we introduce an algorithm to distill structure-preserving motion priors from an autoregressive video tracking model (SAM2) into a bidirectional video diffusion model (CogVideoX). With our method, we train SAM2VideoX, which contains two innovations: (1) a bidirectional feature fusion module that extracts global structure-preserving motion priors from a recurrent model like SAM2; (2) a Local Gram Flow loss that aligns how local features move together. Experiments on VBench and in human studies show that SAM2VideoX delivers consistent gains (+2.60\% on VBench, 21-22\% lower FVD, and 71.4\% human preference) over prior baselines. Specifically, on VBench, we achieve 95.51\%, surpassing REPA (92.91\%) by 2.60\%, and reduce FVD to 360.57, a 21.20\% and 22.46\% improvement over REPA- and LoRA-finetuning, respectively. The project website can be found at this https URL .
Abstract (translated)
现实是在刚性约束和可变形结构之间的舞蹈。对于视频模型而言，这意味着生成既保持保真度又保留结构的运动。尽管在扩散模型方面取得了进展，但产生真实且结构保守的动作仍然具有挑战性，特别是对于像人类和动物这样的连杆和可变形对象来说。仅靠扩大训练数据尚未解决物理上不可能的过渡问题。现有的方法依赖于使用嘈杂的动作表示进行条件设置，例如通过外部不完美的模型提取的光学流或骨架。 为了解决这些挑战，我们引入了一种算法，将结构保守运动先验从自回归视频跟踪模型（SAM2）提炼到双向视频扩散模型（CogVideoX）中。利用我们的方法，我们训练了名为SAM2VideoX的新模型，该模型包含两项创新： 1. 一个双向特征融合模块，可以从像SAM2这样的递归模型中提取全局结构保守运动先验。 2. 局部Gram Flow损失，用于对齐局部特征如何共同移动。 在VBench和人类研究中的实验表明，SAM2VideoX相对于之前的基准线提供了持续的改进（VBench上的+2.60%，FVD降低了21-22%，并且在71.4%的人类偏好度上表现出优势）。具体来说，在VBench上我们达到了95.51%，超越了REPA的92.91%，将FVD降低到了360.57，分别比REPA和LoRA微调改进了21.20% 和 22.46%。 项目的网站可以在[这里](https://this-should-be-the-url-of-the-project)找到。请注意，链接处应填写实际的项目网址。
URL
https://arxiv.org/abs/2512.11792
