Abstract
We introduce Follow-Your-Creation, a novel 4D video creation framework capable of both generating and editing 4D content from a single monocular video input. By leveraging a powerful video inpainting foundation model as a generative prior, we reformulate 4D video creation as a video inpainting task, enabling the model to fill in missing content caused by camera trajectory changes or user edits. To facilitate this, we generate composite masked inpainting video data to effectively fine-tune the model for 4D video generation. Given an input video and its associated camera trajectory, we first perform depth-based point cloud rendering to obtain invisibility masks that indicate the regions that should be completed. Simultaneously, editing masks are introduced to specify user-defined modifications, and these are combined with the invisibility masks to create a composite masks dataset. During training, we randomly sample different types of masks to construct diverse and challenging inpainting scenarios, enhancing the model's generalization and robustness in various 4D editing and generation tasks. To handle temporal consistency under large camera motion, we design a self-iterative tuning strategy that gradually increases the viewing angles during training, where the model is used to generate the next-stage training data after each fine-tuning iteration. Moreover, we introduce a temporal packaging module during inference to enhance generation quality. Our method effectively leverages the prior knowledge of the base model without degrading its original performance, enabling the generation of 4D videos with consistent multi-view coherence. In addition, our approach supports prompt-based content editing, demonstrating strong flexibility and significantly outperforming state-of-the-art methods in both quality and versatility.
Abstract (translated)
我们介绍了一种名为“Follow-Your-Creation”的新型4D视频创作框架，该框架能够从单目视频输入中生成和编辑4D内容。通过利用强大的视频修复基础模型作为生成先验条件，我们将4D视频的创建任务重新表述为一个视频修复任务，从而使模型能够在因相机轨迹变化或用户修改导致的内容缺失时进行填充。为了支持这一过程，我们生成了合成掩码修复视频数据集，以有效地微调模型用于4D视频生成。给定输入视频及其相关联的相机轨迹后，首先通过基于深度的点云渲染获得不可视性掩码，指示需要完成的区域。同时引入编辑掩码来指定用户定义的修改，并将这些与不可视性掩码结合以创建合成掩码数据集。在训练过程中，我们随机选择不同类型的掩码构建多样且具有挑战性的修复场景，从而增强模型在各种4D编辑和生成任务中的泛化能力和鲁棒性。 为了处理大范围相机运动下的时间一致性问题，我们设计了一种自我迭代调优策略，在训练期间逐步增加视角角度。每次微调迭代后，模型用于生成下一阶段的训练数据。此外，在推理过程中引入了时序打包模块以增强生成质量。我们的方法有效利用基础模型的知识而不损害其原始性能，从而生成具有多视图一致性的4D视频。此外，我们的方法支持基于提示的内容编辑，展示了强大的灵活性，并在质量和多功能性方面显著优于现有技术。 总的来说，“Follow-Your-Creation”框架不仅能够生成高质量的4D内容，还能灵活地进行用户定义的修改，适用于各种应用场景。
URL
https://arxiv.org/abs/2506.04590
