Abstract
Video generation is rapidly evolving towards unified audio-video generation. In this paper, we present ALIVE, a generation model that adapts a pretrained Text-to-Video (T2V) model to Sora-style audio-video generation and animation. In particular, the model unlocks the Text-to-Video&Audio (T2VA) and Reference-to-Video&Audio (animation) capabilities compared to the T2V foundation models. To support the audio-visual synchronization and reference animation, we augment the popular MMDiT architecture with a joint audio-video branch which includes TA-CrossAttn for temporally-aligned cross-modal fusion and UniTemp-RoPE for precise audio-visual alignment. Meanwhile, a comprehensive data pipeline consisting of audio-video captioning, quality control, etc., is carefully designed to collect high-quality finetuning data. Additionally, we introduce a new benchmark to perform a comprehensive model test and comparison. After continue pretraining and finetuning on million-level high-quality data, ALIVE demonstrates outstanding performance, consistently outperforming open-source models and matching or surpassing state-of-the-art commercial solutions. With detailed recipes and benchmarks, we hope ALIVE helps the community develop audio-video generation models more efficiently. Official page: this https URL.
Abstract (translated)
视频生成正迅速发展为统一的音视频生成。在本文中，我们介绍了ALIVE模型，这是一个将预训练的文本到视频（Text-to-Video，T2V）模型适应于Sora风格的音视频生成和动画工作的生成模型。特别地，与基础的T2V模型相比，该模型解锁了文本到视频音频（Text-to-Video&Audio, T2VA）以及参考到视频音频（动画）的能力。为了支持视听同步及参考动画，我们增强了流行的MMDiT架构，加入了一个联合音视频分支，其中包括用于时间对齐的跨模态融合技术TA-CrossAttn和用于精确视听对齐的UniTemp-RoPE。 同时，设计了一套全面的数据处理流程，包括音频-视频配字幕、质量控制等步骤，以收集高质量的微调数据。此外，我们还引入了一个新的基准测试来进行全面的模型性能测试与比较。经过百万级高质数据的持续预训练和微调之后，ALIVE展现了卓越的性能，在多个方面超过了开源模型，并且达到了或超过了业内最先进的商业解决方案。 通过详细的实现步骤和基准测试标准，希望ALIVE能帮助社区更高效地开发音视频生成模型。官方页面: [提供链接]
URL
https://arxiv.org/abs/2602.08682
