Abstract
Human communication is multimodal, with speech and gestures tightly coupled, yet most computational methods for generating speech and gestures synthesize them sequentially, weakening synchrony and prosody alignment. We introduce Gelina, a unified framework that jointly synthesizes speech and co-speech gestures from text using interleaved token sequences in a discrete autoregressive backbone, with modality-specific decoders. Gelina supports multi-speaker and multi-style cloning and enables gesture-only synthesis from speech inputs. Subjective and objective evaluations demonstrate competitive speech quality and improved gesture generation over unimodal baselines.
Abstract (translated)
人类的交流是多模态的，言语和手势紧密相连，然而大多数用于生成言语和手势的计算方法都是顺序合成的，这会削弱同步性和语调的一致性。我们介绍了 Gelina，这是一个统一框架，它使用离散自回归主干中的交错令牌序列从文本中联合合成语音和伴随语言的手势，并且配有特定于每种模式的解码器。Gelina 支持多说话人和多种风格的克隆，并能够仅通过语音输入生成手势。主观和客观评估表明，与单模态基线相比，该框架在言语质量方面具有竞争力，在手势生成方面有所改进。
URL
https://arxiv.org/abs/2510.12834
