Abstract
Despite recent progress, video diffusion models still struggle to synthesize realistic videos involving highly dynamic motions or requiring fine-grained motion controllability. A central limitation lies in the scarcity of such examples in commonly used training datasets. To address this, we introduce DynaVid, a video synthesis framework that leverages synthetic motion data in training, which is represented as optical flow and rendered using computer graphics pipelines. This approach offers two key advantages. First, synthetic motion offers diverse motion patterns and precise control signals that are difficult to obtain from real data. Second, unlike rendered videos with artificial appearances, rendered optical flow encodes only motion and is decoupled from appearance, thereby preventing models from reproducing the unnatural look of synthetic videos. Building on this idea, DynaVid adopts a two-stage generation framework: a motion generator first synthesizes motion, and then a motion-guided video generator produces video frames conditioned on that motion. This decoupled formulation enables the model to learn dynamic motion patterns from synthetic data while preserving visual realism from real-world videos. We validate our framework on two challenging scenarios, vigorous human motion generation and extreme camera motion control, where existing datasets are particularly limited. Extensive experiments demonstrate that DynaVid improves the realism and controllability in dynamic motion generation and camera motion control.
Abstract (translated)
尽管近期取得了进展，视频扩散模型在合成涉及高度动态运动或需要细粒度运动可控性的逼真视频时仍面临挑战。其核心限制在于常用训练数据集中此类样本的匮乏。为此，我们提出了DynaVid——一个利用合成运动数据进行训练的视频合成框架，该数据以光流形式表示并通过计算机图形管线渲染。该方法具有两大优势：首先，合成运动能提供多样化的运动模式和精确的控制信号，而这些从真实数据中难以获取；其次，与外观人工的渲染视频不同，渲染光流仅编码运动信息且与外观解耦，从而避免模型复现合成视频的不自然外观。基于此思路，DynaVid采用两阶段生成框架：运动生成器先合成运动，随后运动引导的视频生成器以该运动为条件生成视频帧。这种解耦设计使模型能从合成数据中学习动态运动模式，同时保留真实视频的视觉真实性。我们在两个挑战性场景中验证了该框架——剧烈人体运动生成与极端摄像机运动控制（现有数据集在这些领域尤为匮乏）。大量实验表明，DynaVid提升了动态运动生成与摄像机运动控制的真实性与可控性。
URL
https://arxiv.org/abs/2604.01666
