Abstract
We present Video Motion Graphs, a system designed to generate realistic human motion videos. Using a reference video and conditional signals such as music or motion tags, the system synthesizes new videos by first retrieving video clips with gestures matching the conditions and then generating interpolation frames to seamlessly connect clip boundaries. The core of our approach is HMInterp, a robust Video Frame Interpolation (VFI) model that enables seamless interpolation of discontinuous frames, even for complex motion scenarios like dancing. HMInterp i) employs a dual-branch interpolation approach, combining a Motion Diffusion Model for human skeleton motion interpolation with a diffusion-based video frame interpolation model for final frame generation. ii) adopts condition progressive training to effectively leverage identity strong and weak conditions, such as images and pose. These designs ensure both high video texture quality and accurate motion trajectory. Results show that our Video Motion Graphs outperforms existing generative- and retrieval-based methods for multi-modal conditioned human motion video generation. Project page can be found at this https URL
Abstract (translated)
我们介绍了Video Motion Graphs（VMG）系统，这是一个旨在生成逼真的人体动作视频的创新解决方案。通过使用参考视频以及音乐或动作标签等条件信号，该系统能够合成新的视频：首先检索与给定条件匹配的动作片段，然后生成插值帧以无缝连接这些片段之间的边界。我们方法的核心是HMInterp，这是一种强大的视频帧插值（VFI）模型，它能够在包括舞蹈在内的复杂运动场景中实现不连续帧的平滑插值。 HMInterp具有以下两个关键特性： i) 采用了双分支插值方法，结合了用于人体骨骼动作插值的Motion Diffusion Model和基于扩散的方法进行最终帧生成。 ii) 采用逐步条件训练策略，有效利用身份强弱条件（如图像和姿态），以确保视频纹理质量和准确的动作轨迹。 实验结果表明，在多模态条件下的人体运动视频生成方面，我们的Video Motion Graphs系统优于现有的生成式及检索式的解决方案。有关项目的更多信息可以在以下链接找到：[项目页面](https://this https URL) （请将URL中的"this https URL"替换为实际的项目网址）。
URL
https://arxiv.org/abs/2503.20218
