Abstract
Recent video generation models demonstrate impressive synthesis capabilities but remain limited by single-modality conditioning, constraining their holistic world understanding. This stems from insufficient cross-modal interaction and limited modal diversity for comprehensive world knowledge representation. To address these limitations, we introduce UnityVideo, a unified framework for world-aware video generation that jointly learns across multiple modalities (segmentation masks, human skeletons, DensePose, optical flow, and depth maps) and training paradigms. Our approach features two core components: (1) dynamic noising to unify heterogeneous training paradigms, and (2) a modality switcher with an in-context learner that enables unified processing via modular parameters and contextual learning. We contribute a large-scale unified dataset with 1.3M samples. Through joint optimization, UnityVideo accelerates convergence and significantly enhances zero-shot generalization to unseen data. We demonstrate that UnityVideo achieves superior video quality, consistency, and improved alignment with physical world constraints. Code and data can be found at: this https URL
Abstract (translated)
最近的视频生成模型展示了令人印象深刻的合成能力，但仍然受限于单一模态的条件设定，这限制了它们对世界整体理解的能力。这种局限性源于跨模态互动不足和用于综合世界知识表示的模态多样性有限。为了解决这些问题，我们引入了一个名为UnityVideo的统一框架，该框架旨在通过多模态（包括分割掩码、人体骨架、密集姿态图、光流以及深度图）进行学习，并采用多种训练范式。我们的方法包含两个核心组成部分：(1) 动态噪声处理，用于统一分异的训练范式；(2) 带有上下文学习器的模态切换器，通过模块化参数和上下文学习实现统一处理。 我们贡献了一个大规模的多模态数据集，其中包含超过130万个样本。通过联合优化，UnityVideo加速了收敛速度，并显著提升了对未见过的数据的零样本泛化能力。实验表明，UnityVideo实现了视频质量、一致性和与物理世界约束更好的对齐方面的改进。 代码和数据可在以下链接获取：[此URL](this https URL)
URL
https://arxiv.org/abs/2512.07831
