Abstract
The In-context generation paradigm recently has demonstrated strong power in instructional image editing with both data efficiency and synthesis quality. Nevertheless, shaping such in-context learning for instruction-based video editing is not trivial. Without specifying editing regions, the results can suffer from the problem of inaccurate editing regions and the token interference between editing and non-editing areas during denoising. To address these, we present ReCo, a new instructional video editing paradigm that novelly delves into constraint modeling between editing and non-editing regions during in-context generation. Technically, ReCo width-wise concatenates source and target video for joint denoising. To calibrate video diffusion learning, ReCo capitalizes on two regularization terms, i.e., latent and attention regularization, conducting on one-step backward denoised latents and attention maps, respectively. The former increases the latent discrepancy of the editing region between source and target videos while reducing that of non-editing areas, emphasizing the modification on editing area and alleviating outside unexpected content generation. The latter suppresses the attention of tokens in the editing region to the tokens in counterpart of the source video, thereby mitigating their interference during novel object generation in target video. Furthermore, we propose a large-scale, high-quality video editing dataset, i.e., ReCo-Data, comprising 500K instruction-video pairs to benefit model training. Extensive experiments conducted on four major instruction-based video editing tasks demonstrate the superiority of our proposal.
Abstract (translated)
最近，基于上下文生成的范式在数据效率和合成质量方面展示了强大的图像编辑能力。然而，将这种基于上下文的学习方法应用于指令驱动的视频编辑并非易事。如果没有指定编辑区域，在去噪过程中会遇到不准确的编辑区域问题以及编辑区域与非编辑区域之间令牌干扰的问题。为了解决这些问题，我们提出了ReCo（重新约束），这是一种新的指令式视频编辑范例，它新颖地在上下文生成期间探讨了编辑区域和非编辑区域之间的约束建模。 技术上讲，ReCo 横向连接源视频和目标视频进行联合去噪处理。为了校准视频扩散学习，ReCo 利用了两种正则化项：即潜伏正则化（latent regularization）和注意力正则化（attention regularization），分别作用于一步逆去噪后的潜在特征和注意力图上。前者增加了源视频与目标视频之间编辑区域的潜在差异，同时减少了非编辑区域的潜在差异，强调了对编辑区域进行修改的重要性，并减轻了外部未预期内容生成的问题。后者抑制了编辑区域内令牌对源视频相应位置令牌的关注度，从而在目标视频中的新对象生成过程中减少了它们之间的干扰。 此外，我们还提出了一套大规模、高质量的视频编辑数据集，即ReCo-Data，包含50万条指令-视频配对的数据以支持模型训练。我们在四个主要基于指令的视频编辑任务上进行了一系列实验，证明了我们的方法具有显著优势。
URL
https://arxiv.org/abs/2512.17650
