Abstract
The Text to Audible-Video Generation (TAVG) task involves generating videos with accompanying audio based on text descriptions. Achieving this requires skillful alignment of both audio and video elements. To support research in this field, we have developed a comprehensive Text to Audible-Video Generation Benchmark (TAVGBench), which contains over 1.7 million clips with a total duration of 11.8 thousand hours. We propose an automatic annotation pipeline to ensure each audible video has detailed descriptions for both its audio and video contents. We also introduce the Audio-Visual Harmoni score (AVHScore) to provide a quantitative measure of the alignment between the generated audio and video modalities. Additionally, we present a baseline model for TAVG called TAVDiffusion, which uses a two-stream latent diffusion model to provide a fundamental starting point for further research in this area. We achieve the alignment of audio and video by employing cross-attention and contrastive learning. Through extensive experiments and evaluations on TAVGBench, we demonstrate the effectiveness of our proposed model under both conventional metrics and our proposed metrics.
Abstract (translated)
TAVG（文本到听觉视频生成）任务涉及根据文本描述生成带有音频的视频。要实现这一目标，需要对音频和视频元素进行精确的同步。为了支持该领域的研究，我们开发了一个全面的文本到听觉视频生成基准（TAVGBench），包含超过1700万段时长为11.8万小时的片段。我们提出了一种自动注释管道，以确保每个听觉视频都有其音频和视频内容的详细描述。我们还引入了音频视觉和谐分数（AVHScore）来提供生成音频和视频之间 alignment 的定量测量。此外，我们还推出了TAVDiffusion baseline模型，该模型使用两个流式latent扩散模型为该领域进一步研究提供了一个基本的起点。通过在TAVGBench上进行广泛的实验和评估，我们证明了我们提出的模型在常规指标和提出的指标上都具有有效性。
URL
https://arxiv.org/abs/2404.14381
