Abstract
Unified multimodal models have shown promising results in multimodal content generation and editing but remain largely limited to the image domain. In this work, we present UniVideo, a versatile framework that extends unified modeling to the video domain. UniVideo adopts a dual-stream design, combining a Multimodal Large Language Model (MLLM) for instruction understanding with a Multimodal DiT (MMDiT) for video generation. This design enables accurate interpretation of complex multimodal instructions while preserving visual consistency. Built on this architecture, UniVideo unifies diverse video generation and editing tasks under a single multimodal instruction paradigm and is jointly trained across them. Extensive experiments demonstrate that UniVideo matches or surpasses state-of-the-art task-specific baselines in text/image-to-video generation, in-context video generation and in-context video editing. Notably, the unified design of UniVideo enables two forms of generalization. First, UniVideo supports task composition, such as combining editing with style transfer, by integrating multiple capabilities within a single instruction. Second, even without explicit training on free-form video editing, UniVideo transfers its editing capability from large-scale image editing data to this setting, handling unseen instructions such as green-screening characters or changing materials within a video. Beyond these core capabilities, UniVideo also supports visual-prompt-based video generation, where the MLLM interprets visual prompts and guides the MMDiT during synthesis. To foster future research, we will release our model and code.
Abstract (translated)
统一的多模态模型在多模态内容生成和编辑方面显示出了令人鼓舞的结果，但仍然主要局限于图像领域。在这项工作中，我们介绍了UniVideo，这是一个灵活的框架，它将统一建模扩展到了视频领域。UniVideo采用了双流设计，结合了用于指令理解的多模态大型语言模型（MLLM）与用于视频生成的多模态DiT（MMDiT）。这种设计能够准确解释复杂的多模态指令，并保持视觉一致性。基于这一架构，UniVideo将多样化的视频生成和编辑任务统一在一个单一的多模态指令范式下，并在这些任务之间进行联合训练。大量的实验表明，UniVideo在文本/图像到视频生成、上下文视频生成以及上下文视频编辑方面达到了或超越了最先进的特定任务基线。 值得注意的是，UniVideo的统一设计能够实现两种形式的泛化：首先，UniVideo支持任务组合，如将编辑与风格迁移相结合，在单一指令中集成多种能力。其次，即使没有在自由形式视频编辑上进行明确训练，UniVideo也能将其编辑能力从大规模图像编辑数据迁移到这种环境中，处理诸如绿幕人物或改变视频中材质等未见过的指令。 除了这些核心功能外，UniVideo还支持基于视觉提示的视频生成，在这种模式下，MLLM解释视觉提示并指导MMDiT在合成过程中。为了促进未来的研究，我们将发布我们的模型和代码。
URL
https://arxiv.org/abs/2510.08377
