Abstract
Image-driven video editing aims to propagate edit contents from the modified first frame to the rest frames. The existing methods usually invert the source video to noise using a pre-trained image-to-video (I2V) model and then guide the sampling process using the edited first frame. Generally, a popular choice for maintaining motion and layout from the source video is intervening in the denoising process by injecting attention during reconstruction. However, such injection often leads to unsatisfactory results, where excessive injection leads to conflicting semantics from the source video while insufficient injection brings limited source representation. Recognizing this, we propose an Editing-awaRE (REE) injection method to modulate injection intensity of each token. Specifically, we first compute the pixel difference between the source and edited first frame to form a corresponding editing mask. Next, we track the editing area throughout the entire video by using optical flow to warp the first-frame mask. Then, editing-aware feature injection intensity for each token is generated accordingly, where injection is not conducted on editing areas. Building upon REE injection, we further propose a zero-shot image-driven video editing framework with recent-emerging rectified-Flow models, dubbed FREE-Edit. Without fine-tuning or training, our FREE-Edit demonstrates effectiveness in various image-driven video editing scenarios, showing its capability to produce higher-quality outputs compared with existing techniques. Project page: this https URL.
Abstract (translated)
图像驱动的视频编辑旨在将修改后的第一帧上的编辑内容传播到其余所有帧上。现有方法通常会使用预训练的图像到视频（I2V）模型将源视频逆向转换为噪声，然后通过编辑过的首帧来引导采样过程。为了从源视频中保持运动和布局的一致性，一种常见的做法是在去噪过程中注入注意力以进行重建。然而，这种注入方式往往会导致不尽人意的结果：过度注入会使得原始视频中的语义与编辑内容产生冲突，而不足的注入则无法充分表示源视频的内容。 为了应对这一挑战，我们提出了一种名为Editing-awaRE（REE）的注入方法，该方法可以调节每个标记的注入强度。具体来说，首先计算源帧和编辑后的第一帧之间的像素差异，以形成对应的编辑掩码。接着，通过使用光学流技术来变形首帧掩码，我们可以跟踪整个视频中的编辑区域。根据这一过程生成了针对每个标记的、具有编辑意识的功能注入强度，在此过程中不会在编辑区域内进行任何功能注入。 基于REE注入方法，我们进一步提出了一种零样本图像驱动的视频编辑框架——FREE-Edit，并结合了最近出现的矫正流模型（rectified-Flow models）。无需微调或训练，我们的FREE-Edit框架能够在各种图像驱动的视频编辑场景中展现其有效性，并且能够产生比现有技术更高的质量输出。 项目页面：[请在此处替换为实际链接]
URL
https://arxiv.org/abs/2603.01164
