Paper Reading AI Learner

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

2024-03-16 04:40:10
Fan Zhang, Zhaohan Wang, Xin Lyu, Siyuan Zhao, Mengjian Li, Weidong Geng, Naye Ji, Hui Du, Fuxing Gao, Hao Wu, Shunman Li

Abstract

Speech-driven gesture generation is an emerging field within virtual human creation. However, a significant challenge lies in accurately determining and processing the multitude of input features (such as acoustic, semantic, emotional, personality, and even subtle unknown features). Traditional approaches, reliant on various explicit feature inputs and complex multimodal processing, constrain the expressiveness of resulting gestures and limit their applicability. To address these challenges, we present Persona-Gestor, a novel end-to-end generative model designed to generate highly personalized 3D full-body gestures solely relying on raw speech audio. The model combines a fuzzy feature extractor and a non-autoregressive Adaptive Layer Normalization (AdaLN) transformer diffusion architecture. The fuzzy feature extractor harnesses a fuzzy inference strategy that automatically infers implicit, continuous fuzzy features. These fuzzy features, represented as a unified latent feature, are fed into the AdaLN transformer. The AdaLN transformer introduces a conditional mechanism that applies a uniform function across all tokens, thereby effectively modeling the correlation between the fuzzy features and the gesture sequence. This module ensures a high level of gesture-speech synchronization while preserving naturalness. Finally, we employ the diffusion model to train and infer various gestures. Extensive subjective and objective evaluations on the Trinity, ZEGGS, and BEAT datasets confirm our model's superior performance to the current state-of-the-art approaches. Persona-Gestor improves the system's usability and generalization capabilities, setting a new benchmark in speech-driven gesture synthesis and broadening the horizon for virtual human technology. Supplementary videos and code can be accessed at this https URL

Abstract (translated)

演讲驱动的手势生成是一个新兴的虚拟人类创造领域。然而,准确确定和处理海量的输入特征(如音频、语义、情感、个性化和甚至微妙的未知特征)是一个具有挑战性的任务。传统方法,依赖各种显性特征输入和复杂的跨模态处理,限制了生成手势的表现力,并限制了它们的适用性。为了应对这些挑战,我们提出了Persona-Gestor,一种仅依赖原始语音音频生成高度个性化的3D全身手势的新端到端生成模型。该模型结合了模糊特征提取器和无自回归自适应层归一化(AdaLN)变换器扩散架构。模糊特征提取器利用模糊推理策略自动推断隐含的连续模糊特征。这些模糊特征以统一的中间特征的形式输入到AdaLN变换器中。AdaLN变换器引入了一个条件机制,在所有标记符上应用一个统一函数,从而有效地建模模糊特征与手势序列之间的相关性。这个模块确保了高水平的手势与语音同步,同时保留了自然性。最后,我们使用扩散模型来训练和推理各种手势。对Trinity、ZEGGS和BEAT数据集的广泛主观和客观评估证实了我们的模型在现有技术水平上具有卓越性能。Persona-Gestor提高了系统的可用性和扩展能力,为手势驱动的虚拟人类合成树立了新的基准,并为虚拟人类技术的发展拓展了更广阔的空间。附加的视频和代码可以通过这个链接访问:https://

URL

https://arxiv.org/abs/2403.10805

PDF

https://arxiv.org/pdf/2403.10805.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot