Abstract
This paper presents DreamDance, a novel character art animation framework capable of producing stable, consistent character and scene motion conditioned on precise camera trajectories. To achieve this, we re-formulate the animation task as two inpainting-based steps: Camera-aware Scene Inpainting and Pose-aware Video Inpainting. The first step leverages a pre-trained image inpainting model to generate multi-view scene images from the reference art and optimizes a stable large-scale Gaussian field, which enables coarse background video rendering with camera trajectories. However, the rendered video is rough and only conveys scene motion. To resolve this, the second step trains a pose-aware video inpainting model that injects the dynamic character into the scene video while enhancing background quality. Specifically, this model is a DiT-based video generation model with a gating strategy that adaptively integrates the character's appearance and pose information into the base background video. Through extensive experiments, we demonstrate the effectiveness and generalizability of DreamDance, producing high-quality and consistent character animations with remarkable camera dynamics.
Abstract (translated)
本文介绍了DreamDance，这是一种新颖的角色艺术动画框架，能够根据精确的相机轨迹生成稳定且一致的角色和场景运动。为了实现这一目标，我们将动画任务重新表述为两个基于修复的方法步骤：感知相机的场景修复（Camera-aware Scene Inpainting）和感知姿态的视频修复（Pose-aware Video Inpainting）。 第一个步骤利用预训练的图像修复模型从参考艺术作品生成多视角场景图像，并优化一个稳定的大型高斯场，从而启用带有相机轨迹的粗略背景视频渲染。然而，渲染出的视频粗糙且仅传达场景运动信息。为了改善这一点，第二个步骤通过训练一种感知姿态的视频修复模型来解决这一问题，该模型将动态角色注入到场景视频中，并提升背景质量。具体而言，这是一种基于DiT（Diffusion Transformer）的视频生成模型，采用了一种门控策略，能够根据自适应地整合角色的外观和姿态信息进基础背景视频。 通过广泛的实验，我们证明了DreamDance的有效性和泛化能力，它能生成高质量且一致的角色动画，并具有显著的相机动态特性。
URL
https://arxiv.org/abs/2505.24733
