Abstract
We present ReCoM, an efficient framework for generating high-fidelity and generalizable human body motions synchronized with speech. The core innovation lies in the Recurrent Embedded Transformer (RET), which integrates Dynamic Embedding Regularization (DER) into a Vision Transformer (ViT) core architecture to explicitly model co-speech motion dynamics. This architecture enables joint spatial-temporal dependency modeling, thereby enhancing gesture naturalness and fidelity through coherent motion synthesis. To enhance model robustness, we incorporate the proposed DER strategy, which equips the model with dual capabilities of noise resistance and cross-domain generalization, thereby improving the naturalness and fluency of zero-shot motion generation for unseen speech inputs. To mitigate inherent limitations of autoregressive inference, including error accumulation and limited self-correction, we propose an iterative reconstruction inference (IRI) strategy. IRI refines motion sequences via cyclic pose reconstruction, driven by two key components: (1) classifier-free guidance improves distribution alignment between generated and real gestures without auxiliary supervision, and (2) a temporal smoothing process eliminates abrupt inter-frame transitions while ensuring kinematic continuity. Extensive experiments on benchmark datasets validate ReCoM's effectiveness, achieving state-of-the-art performance across metrics. Notably, it reduces the Fréchet Gesture Distance (FGD) from 18.70 to 2.48, demonstrating an 86.7% improvement in motion realism. Our project page is this https URL.
Abstract (translated)
我们提出了一种名为ReCoM的高效框架，用于生成与语音同步的高度真实和通用性的人体运动。该框架的核心创新在于引入了循环嵌入变换器（Recurrent Embedded Transformer, RET），它将动态嵌入正则化（Dynamic Embedding Regularization, DER）整合到视觉变换器（Vision Transformer, ViT）的架构中，以显式地建模与言语同步的身体运动动力学。这种架构通过联合空间-时间依赖性建模来增强手势的真实性和连贯性。 为了提升模型的鲁棒性，我们引入了DER策略，使模型同时具备抗噪能力和跨域泛化能力，从而改善零样本运动生成（即对于未见过的语音输入）的真实感和流畅度。为了解决自回归推理固有的问题，如误差积累和自我修正能力有限的问题，我们提出了一种迭代重构推理（Iterative Reconstruction Inference, IRI）策略。IRI通过周期性姿势重建来细化运动序列，并由两个关键组件驱动：1) 无分类器引导提高了生成手势与真实手势之间的分布一致性，无需辅助监督；2) 时间平滑过程消除了帧间突然的过渡变化，同时确保了动力学连续性。 在基准数据集上的广泛实验验证了ReCoM的有效性，并实现了跨多个指标的最佳性能。尤其值得注意的是，它将Fréchet动作距离（FGD）从18.70降低到2.48，运动真实性提高了86.7%。我们的项目页面在此网址：[此URL]。
URL
https://arxiv.org/abs/2503.21847
