Abstract
The emergence of Diffusion Transformers (DiT) has brought significant advancements to video generation, especially in text-to-video and image-to-video tasks. Although video generation is widely applied in various fields, most existing models are limited to single scenarios and cannot perform diverse video generation and editing through dynamic content manipulation. We propose OmniV2V, a video model capable of generating and editing videos across different scenarios based on various operations, including: object movement, object addition, mask-guided video edit, try-on, inpainting, outpainting, human animation, and controllable character video synthesis. We explore a unified dynamic content manipulation injection module, which effectively integrates the requirements of the above tasks. In addition, we design a visual-text instruction module based on LLaVA, enabling the model to effectively understand the correspondence between visual content and instructions. Furthermore, we build a comprehensive multi-task data processing system. Since there is data overlap among various tasks, this system can efficiently provide data augmentation. Using this system, we construct a multi-type, multi-scenario OmniV2V dataset and its corresponding OmniV2V-Test benchmark. Extensive experiments show that OmniV2V works as well as, and sometimes better than, the best existing open-source and commercial models for many video generation and editing tasks.
Abstract (translated)
Diffusion Transformers（DiT）的出现为视频生成领域带来了显著的进步，特别是在文本到视频和图像到视频的任务中。尽管视频生成技术在各个领域的应用广泛，但大多数现有的模型仅限于单一场景的应用，并且无法通过动态内容操作来进行多样化的视频生成与编辑。我们提出了OmniV2V，这是一种能够在不同场景下根据各种操作（如物体移动、对象添加、基于蒙版的视频编辑、试穿、修复、扩展、人类动画和可控角色视频合成）进行视频生成和编辑的模型。我们探索了一个统一的动态内容操纵注入模块，该模块能够有效整合上述任务的需求。此外，我们设计了一种基于LLaVA的视觉-文本指令模块，使模型能够有效地理解视觉内容与指令之间的对应关系。另外，我们构建了全面的多任务数据处理系统。由于各任务之间存在数据重叠，这一系统可以高效地提供数据增强。利用这套系统，我们建立了一个多类型、多场景的OmniV2V数据集及其对应的OmniV2V-Test基准测试集。大量的实验表明，对于许多视频生成和编辑的任务，OmniV2V的表现与当前最佳开源及商用模型相当甚至更优。
URL
https://arxiv.org/abs/2506.01801
