Abstract
The intrinsic link between facial motion and speech is often overlooked in generative modeling, where talking head synthesis and text-to-speech (TTS) are typically addressed as separate tasks. This paper introduces JAM-Flow, a unified framework to simultaneously synthesize and condition on both facial motion and speech. Our approach leverages flow matching and a novel Multi-Modal Diffusion Transformer (MM-DiT) architecture, integrating specialized Motion-DiT and Audio-DiT modules. These are coupled via selective joint attention layers and incorporate key architectural choices, such as temporally aligned positional embeddings and localized joint attention masking, to enable effective cross-modal interaction while preserving modality-specific strengths. Trained with an inpainting-style objective, JAM-Flow supports a wide array of conditioning inputs-including text, reference audio, and reference motion-facilitating tasks such as synchronized talking head generation from text, audio-driven animation, and much more, within a single, coherent model. JAM-Flow significantly advances multi-modal generative modeling by providing a practical solution for holistic audio-visual synthesis. project page: this https URL
Abstract (translated)
在生成模型中，面部动作与语音之间的内在联系常常被忽视，通常会将说话头部合成（Talking Head Synthesis）和文本转语音（Text-to-Speech, TTS）视为两个独立的任务。本文介绍了一种名为JAM-Flow的统一框架，它可以同时综合并基于面部运动和语音进行条件设定。我们的方法利用了流匹配技术和一种新颖的多模态扩散变压器（Multi-Modal Diffusion Transformer, MM-DiT）架构，该架构结合了专门设计的动作-DiT和音频-DiT模块。这些模块通过选择性联合注意力层相互连接，并采用了关键的架构决策，如时间对齐的位置嵌入和局部联合注意力掩码，以实现有效的跨模态交互并保留特定模式的优势。 JAM-Flow采用一种补全风格的目标进行训练，支持多种条件输入形式——包括文本、参考音频以及参考动作——这使它能够在一个单一的连贯模型中处理诸如从文本生成同步说话头部、由音频驱动的动画等任务，并且还有更多可能的应用场景。通过为整体音视频合成提供实用解决方案，JAM-Flow显著推进了多模态生成建模技术的发展。 项目主页：[此 URL](https://this-url.com/)（请将链接替换为您提供的实际网址）
URL
https://arxiv.org/abs/2506.23552
