Abstract
Video editing methods based on diffusion models that rely solely on a text prompt for the edit are hindered by the limited expressive power of text prompts. Thus, incorporating a reference target image as a visual guide becomes desirable for precise control over edit. Also, most existing methods struggle to accurately edit a video when the shape and size of the object in the target image differ from the source object. To address these challenges, we propose "GenVideo" for editing videos leveraging target-image aware T2I models. Our approach handles edits with target objects of varying shapes and sizes while maintaining the temporal consistency of the edit using our novel target and shape aware InvEdit masks. Further, we propose a novel target-image aware latent noise correction strategy during inference to improve the temporal consistency of the edits. Experimental analyses indicate that GenVideo can effectively handle edits with objects of varying shapes, where existing approaches fail.
Abstract (translated)
基于扩散模型的视频编辑方法，仅依赖文本提示进行编辑，受到文本提示表达能力的限制。因此，将参考目标图像作为视觉指南变得具有吸引力，以便更精确地控制编辑。同时，大多数现有方法在处理目标图像中物体形状和大小与源对象不匹配时，准确性都会受到影响。为解决这些挑战，我们提出了“GenVideo”，一种利用目标图像意识到的T2I模型编辑视频的方法。我们的方法能够处理具有不同形状和大小的目标对象的编辑，同时通过我们新颖的目标和形状意识InvEdit掩码保持编辑的时间一致性。此外，在推理过程中，我们提出了一种新的目标图像意识延迟噪声修复策略，以提高编辑的时间一致性。实验分析结果表明，GenVideo可以有效地处理具有不同形状的物体，而现有方法在此方面都存在局限性。
URL
https://arxiv.org/abs/2404.12541
