Abstract
Speech-driven gesture generation is an emerging field within virtual human creation. However, a significant challenge lies in accurately determining and processing the multitude of input features (such as acoustic, semantic, emotional, personality, and even subtle unknown features). Traditional approaches, reliant on various explicit feature inputs and complex multimodal processing, constrain the expressiveness of resulting gestures and limit their applicability. To address these challenges, we present Persona-Gestor, a novel end-to-end generative model designed to generate highly personalized 3D full-body gestures solely relying on raw speech audio. The model combines a fuzzy feature extractor and a non-autoregressive Adaptive Layer Normalization (AdaLN) transformer diffusion architecture. The fuzzy feature extractor harnesses a fuzzy inference strategy that automatically infers implicit, continuous fuzzy features. These fuzzy features, represented as a unified latent feature, are fed into the AdaLN transformer. The AdaLN transformer introduces a conditional mechanism that applies a uniform function across all tokens, thereby effectively modeling the correlation between the fuzzy features and the gesture sequence. This module ensures a high level of gesture-speech synchronization while preserving naturalness. Finally, we employ the diffusion model to train and infer various gestures. Extensive subjective and objective evaluations on the Trinity, ZEGGS, and BEAT datasets confirm our model's superior performance to the current state-of-the-art approaches. Persona-Gestor improves the system's usability and generalization capabilities, setting a new benchmark in speech-driven gesture synthesis and broadening the horizon for virtual human technology. Supplementary videos and code can be accessed at this https URL
Abstract (translated)
演讲驱动的手势生成是一个新兴的虚拟人类创造领域。然而，准确确定和处理海量的输入特征（如音频、语义、情感、个性化和甚至微妙的未知特征）是一个具有挑战性的任务。传统方法，依赖各种显性特征输入和复杂的跨模态处理，限制了生成手势的表现力，并限制了它们的适用性。为了应对这些挑战，我们提出了Persona-Gestor，一种仅依赖原始语音音频生成高度个性化的3D全身手势的新端到端生成模型。该模型结合了模糊特征提取器和无自回归自适应层归一化（AdaLN）变换器扩散架构。模糊特征提取器利用模糊推理策略自动推断隐含的连续模糊特征。这些模糊特征以统一的中间特征的形式输入到AdaLN变换器中。AdaLN变换器引入了一个条件机制，在所有标记符上应用一个统一函数，从而有效地建模模糊特征与手势序列之间的相关性。这个模块确保了高水平的手势与语音同步，同时保留了自然性。最后，我们使用扩散模型来训练和推理各种手势。对Trinity、ZEGGS和BEAT数据集的广泛主观和客观评估证实了我们的模型在现有技术水平上具有卓越性能。Persona-Gestor提高了系统的可用性和扩展能力，为手势驱动的虚拟人类合成树立了新的基准，并为虚拟人类技术的发展拓展了更广阔的空间。附加的视频和代码可以通过这个链接访问：https://
URL
https://arxiv.org/abs/2403.10805
