Abstract
Instruction-guided generative models, especially those using text-to-image (T2I) and text-to-video (T2V) diffusion frameworks, have advanced the field of content editing in recent years. To extend these capabilities to 4D scene, we introduce a progressive sampling framework for 4D editing (PSF-4D) that ensures temporal and multi-view consistency by intuitively controlling the noise initialization during forward diffusion. For temporal coherence, we design a correlated Gaussian noise structure that links frames over time, allowing each frame to depend meaningfully on prior frames. Additionally, to ensure spatial consistency across views, we implement a cross-view noise model, which uses shared and independent noise components to balance commonalities and distinct details among different views. To further enhance spatial coherence, PSF-4D incorporates view-consistent iterative refinement, embedding view-aware information into the denoising process to ensure aligned edits across frames and views. Our approach enables high-quality 4D editing without relying on external models, addressing key challenges in previous methods. Through extensive evaluation on multiple benchmarks and multiple editing aspects (e.g., style transfer, multi-attribute editing, object removal, local editing, etc.), we show the effectiveness of our proposed method. Experimental results demonstrate that our proposed method outperforms state-of-the-art 4D editing methods in diverse benchmarks.
Abstract (translated)
指令引导生成模型，尤其是使用文本到图像（T2I）和文本到视频（T2V）扩散框架的那些，在近年来推动了内容编辑领域的发展。为了将这些能力扩展到4D场景中，我们引入了一种渐进式采样框架——用于4D编辑的PSF-4D（Progressive Sampling Framework for 4D），通过直观地控制前向扩散过程中的噪声初始化来确保时间一致性和多视角一致性。 对于时间连贯性，我们设计了一个相关高斯噪声结构，该结构将帧按时间顺序链接起来，使得每一帧都能有意义地依赖于先前的帧。为了确保不同视角间的空间一致性，我们实现了一个跨视角噪声模型，它使用共享和独立的噪声成分来平衡各视角之间的共同点与独特细节。为增强空间连贯性，PSF-4D还集成了视图一致性的迭代细化过程，在去噪过程中嵌入了视图感知信息，以确保编辑在不同帧和视角之间的一致性和对齐。 我们的方法能够在不依赖外部模型的情况下实现高质量的4D编辑，解决了以前方法中的关键挑战。通过在多个基准测试和多种编辑方面（如风格转移、多属性编辑、物体移除、局部编辑等）进行广泛评估，我们展示了所提出方法的有效性。实验结果表明，在各种基准测试中，我们的方法优于现有的最先进的4D编辑技术。
URL
https://arxiv.org/abs/2503.11044
