Abstract
3D animation aims to generate a 3D animated video from an input image and a target 3D motion sequence. Recent advances in image-to-3D models enable the creation of animations directly from user-hand drawings. Distinguished from conventional 3D animation, drawing-based 3D animation is crucial to preserve artist's unique style properties, such as rough contours and distinct stroke patterns. However, recent methods still exhibit quality deterioration in style properties, especially under occlusions caused by overlapping body parts, leading to contour flickering and stroke blurring. This occurs due to a `stylization pose gap' between training and inference in stylization networks designed to preserve drawing styles in drawing-based 3D animation systems. The stylization pose gap denotes that input target poses used to train the stylization network are always in occlusion-free poses, while target poses encountered in an inference include diverse occlusions under dynamic motions. To this end, we propose Occlusion-robust Stylization Framework (OSF) for drawing-based 3D animation. We found that while employing object's edge can be effective input prior for guiding stylization, it becomes notably inaccurate when occlusions occur at inference. Thus, our proposed OSF provides occlusion-robust edge guidance for stylization network using optical flow, ensuring a consistent stylization even under occlusions. Furthermore, OSF operates in a single run instead of the previous two-stage method, achieving 2.4x faster inference and 2.1x less memory.
Abstract (translated)
3D动画的目标是从输入图像和目标3D运动序列生成3D动画视频。最近在图像到3D模型方面的发展使得能够直接从用户手绘的草图创建动画成为可能。与传统的3D动画相比，基于绘画的3D动画对于保留艺术家独特的风格属性（如粗糙轮廓和独特的笔触模式）至关重要。然而，尽管取得了这些进展，目前的方法仍然会在风格属性上表现出质量下降的问题，尤其是在由重叠身体部位导致遮挡时，会导致轮廓闪烁和笔迹模糊。这种现象是由于基于绘画的3D动画系统中用于保留绘制样式的风格化网络在训练与推理阶段之间存在的“姿态差距”（stylization pose gap）所引起的。这个姿态差距表示，在进行模型训练时使用的输入目标姿势总是没有遮挡的情况，而在进行推断时遇到的目标姿势则包含由于动态动作而产生的各种遮挡情况。 为此，我们提出了一种名为抗遮挡风格化框架（Occlusion-robust Stylization Framework, OSF）的方法来解决基于绘画的3D动画的问题。我们发现，虽然使用对象边缘作为引导风格化的有效输入先验是有效的，但在推断过程中发生遮挡时其准确性会显著降低。因此，我们的OSF通过利用光流为风格化网络提供抗遮挡的边缘指引，确保即使在存在遮挡的情况下也能保持一致的风格化效果。 此外，与先前采用的两阶段方法不同，OSF可以在单次运行中完成任务，从而实现了2.4倍更快的推理速度和2.1倍更少的记忆消耗。
URL
https://arxiv.org/abs/2508.00398
