Abstract
In-context image editing aims to modify images based on a contextual sequence comprising text and previously generated images. Existing methods typically depend on task-specific pipelines and expert models (e.g., segmentation and inpainting) to curate training data. In this work, we explore whether an in-context image editing model can be learned directly from videos. We introduce a scalable approach to annotate videos as interleaved multimodal sequences. To effectively learn from this data, we design a block-causal diffusion transformer trained on three proxy tasks: next-image prediction, current segmentation prediction, and next-segmentation prediction. Additionally, we propose a novel multi-turn image editing benchmark to advance research in this area. Extensive experiments demonstrate that our model exhibits strong in-context image editing capabilities and achieves state-of-the-art results on two multi-turn image editing benchmarks. Despite being trained exclusively on videos, our model also shows promising abilities in multi-concept composition, story generation, and chain-of-editing applications.
Abstract (translated)
在上下文中的图像编辑旨在根据包括文本和先前生成的图像在内的上下文序列对图像进行修改。现有的方法通常依赖于特定任务的工作流程和专家模型（例如分割和修复）来整理训练数据。在这项工作中，我们探讨了直接从视频中学习一个上下文图像编辑模型的可能性。我们引入了一种可扩展的方法，将视频注释为交错的多模态序列。为了有效地利用这些数据进行学习，我们设计了一个基于块因果扩散变换器，并针对三个代理任务进行了训练：下一帧预测、当前分割预测和下一帧分割预测。此外，我们还提出了一项新颖的多轮图像编辑基准测试，以推动该领域的研究进展。广泛的实验表明，我们的模型展现了强大的上下文图像编辑能力，在两个多轮图像编辑基准上取得了最先进的成果。尽管仅通过视频进行训练，但我们的模型在多概念组合、故事生成和编辑链应用方面也展现出了令人鼓舞的能力。
URL
https://arxiv.org/abs/2506.10941
