Abstract
With recent advances in Multimodal Large Language Models (MLLMs) showing strong visual understanding and reasoning, interest is growing in using them to improve the editing performance of diffusion models. Despite rapid progress, most studies lack an in-depth analysis of MLLM design choices. Moreover, the integration of MLLMs and diffusion models remains an open challenge in some difficult tasks, such as video editing. In this paper, we present InstructX, a unified framework for image and video editing. Specifically, we conduct a comprehensive study on integrating MLLMs and diffusion models for instruction-driven editing across diverse tasks. Building on this study, we analyze the cooperation and distinction between images and videos in unified modeling. (1) We show that training on image data can lead to emergent video editing capabilities without explicit supervision, thereby alleviating the constraints imposed by scarce video training data. (2) By incorporating modality-specific MLLM features, our approach effectively unifies image and video editing tasks within a single model. Extensive experiments demonstrate that our method can handle a broad range of image and video editing tasks and achieves state-of-the-art performance.
Abstract (translated)
随着多模态大型语言模型（MLLMs）在视觉理解和推理方面取得的显著进展，人们对利用这些模型来提升扩散模型编辑性能的兴趣日益增长。尽管取得了快速进步，但大多数研究缺乏对MLLM设计选择的深入分析。此外，在一些困难任务中，如视频编辑，将MLLM与扩散模型相结合仍然是一个开放性挑战。在本文中，我们提出了InstructX，这是一个用于图像和视频编辑的统一框架。具体来说，我们在各种任务上进行了关于集成MLLM和扩散模型以进行指令驱动编辑的全面研究。在此基础上，我们分析了在统一分模态建模中，图像与视频之间的合作与区别。 （1）我们展示了在仅基于图像数据训练的情况下，可以产生不需要显式监督的视频编辑能力，从而缓解由于缺乏大量视频训练数据而带来的限制。 （2）通过引入特定于模态的MLLM特征，我们的方法能够在一个模型中有效地统一处理图像和视频编辑任务。大量的实验表明，我们提出的方法能够处理广泛的图像和视频编辑任务，并且在性能上达到了最先进的水平。
URL
https://arxiv.org/abs/2510.08485
