Abstract
Audio-driven portrait animation, which synthesizes realistic videos from reference images using audio signals, faces significant challenges in real-time generation of high-fidelity, temporally coherent animations. While recent diffusion-based methods improve generation quality by integrating audio into denoising processes, their reliance on frame-by-frame UNet architectures introduces prohibitive latency and struggles with temporal consistency. This paper introduces MirrorMe, a real-time, controllable framework built on the LTX video model, a diffusion transformer that compresses video spatially and temporally for efficient latent space denoising. To address LTX's trade-offs between compression and semantic fidelity, we propose three innovations: 1. A reference identity injection mechanism via VAE-encoded image concatenation and self-attention, ensuring identity consistency; 2. A causal audio encoder and adapter tailored to LTX's temporal structure, enabling precise audio-expression synchronization; and 3. A progressive training strategy combining close-up facial training, half-body synthesis with facial masking, and hand pose integration for enhanced gesture control. Extensive experiments on the EMTD Benchmark demonstrate MirrorMe's state-of-the-art performance in fidelity, lip-sync accuracy, and temporal stability.
Abstract (translated)
音频驱动的肖像动画技术通过使用音频信号从参考图像合成逼真的视频，但面临着实时生成高质量、时间连贯动画的重大挑战。虽然最近基于扩散的方法通过将音频集成到去噪过程中提高了生成质量，但是它们依赖于帧对帧UNet架构，这引入了高昂的延迟，并且难以实现时间一致性。本文介绍了一种名为MirrorMe的框架，它建立在LTX视频模型之上，这是一种扩散变换器，该变换器可以在空间和时间上压缩视频以进行高效的潜在空间去噪处理。为了应对LTX在压缩与语义保真度之间的权衡问题，我们提出了三项创新： 1. 通过VAE编码图像的拼接和自我注意机制引入参考身份注入机制，确保身份一致性。 2. 设计了一种因果音频编码器和适配器，以适应LTX的时间结构，并实现精确的声音表情同步。 3. 采用一种渐进式训练策略，结合面部特写训练、半身合成（带有面部遮罩）以及手部姿态整合，进一步提高手势控制能力。 在EMTD基准测试上的大量实验表明，MirrorMe在保真度、唇形同步准确性及时间稳定性方面均表现出最先进的性能。
URL
https://arxiv.org/abs/2506.22065
