Abstract
Text-to-video diffusion models have enabled high-quality video synthesis, yet often fail to generate temporally coherent and physically plausible motion. A key reason is the models' insufficient understanding of complex motions that natural videos often entail. Recent works tackle this problem by aligning diffusion model features with those from pretrained video encoders. However, these encoders mix video appearance and dynamics into entangled features, limiting the benefit of such alignment. In this paper, we propose a motion-centric alignment framework that learns a disentangled motion subspace from a pretrained video encoder. This subspace is optimized to predict ground-truth optical flow, ensuring it captures true motion dynamics. We then align the latent features of a text-to-video diffusion model to this new subspace, enabling the generative model to internalize motion knowledge and generate more plausible videos. Our method improves the physical commonsense in a state-of-the-art video diffusion model, while preserving adherence to textual prompts, as evidenced by empirical evaluations on VideoPhy, VideoPhy2, VBench, and VBench-2.0, along with a user study.
Abstract (translated)
文本到视频的扩散模型已经实现了高质量的视频合成，然而它们往往难以生成时间上连贯且物理上合理的运动。主要原因在于这些模型缺乏对自然视频中常见复杂运动的理解。最近的一些研究通过将扩散模型特征与预训练的视频编码器特征对齐来解决这一问题。然而，这种方法会混合视频外观和动态信息，导致提取出的特征纠缠在一起，从而限制了这种对齐方法的效果。 在本文中，我们提出了一种以运动为中心的对齐框架，该框架从预训练的视频编码器中学得一个解耦的运动子空间。这个子空间经过优化可以预测真实的光流，确保它捕捉到了真正的运动动态。然后我们将文本到视频扩散模型中的潜在特征与新的子空间进行对齐，使生成模型能够内化有关运动的知识，并产生更加合理的视频。 我们的方法在不牺牲对文本提示的遵从性的前提下，改进了最先进的视频扩散模型中物理常识的表现力。这一点通过在VideoPhy、VideoPhy2、VBench和VBench-2.0上的实证评估以及用户研究得到了验证。
URL
https://arxiv.org/abs/2510.19022
