Abstract
We present SentiAvatar, a framework for building expressive interactive 3D digital humans, and use it to create SuSu, a virtual character that speaks, gestures, and emotes in real time. Achieving such a system remains challenging, as it requires jointly addressing three key problems: the lack of large-scale, high-quality multimodal data, robust semantic-to-motion mapping, and fine-grained frame-level motion-prosody synchronization. To solve these problems, first, we build SuSuInterActs (21K clips, 37 hours), a dialogue corpus captured via optical motion capture around a single character with synchronized speech, full-body motion, and facial expressions. Second, we pre-train a Motion Foundation Model on 200K+ motion sequences, equipping it with rich action priors that go well beyond the conversation. We then propose an audio-aware plan-then-infill architecture that decouples sentence-level semantic planning from frame-level prosody-driven interpolation, so that generated motions are both semantically appropriate and rhythmically aligned with speech. Experiments show that SentiAvatar achieves state-of-the-art on both SuSuInterActs (R@1 43.64%, nearly 2 times the best baseline) and BEATv2 (FGD 4.941, BC 8.078), producing 6s of output in 0.3s with unlimited multi-turn streaming. The source code, model, and dataset are available at this https URL.
Abstract (translated)
我们提出了SentiAvatar，一个用于构建富有表现力的交互式3D数字人框架，并利用它创建了虚拟角色SuSu，该角色能够实时说话、做手势并表达情感。实现这样的系统仍具有挑战性，因为它需要共同解决三个关键问题：缺乏大规模、高质量的多模态数据，鲁棒的语义到运动映射，以及细粒度的帧级运动-韵律同步。为解决这些问题，首先，我们构建了SuSuInterActs（21,000个片段，37小时），这是一个通过光学运动捕捉采集的对话语料库，围绕单一角色录制，并同步了语音、全身运动和面部表情。其次，我们在200,000多个运动序列上预训练了一个运动基础模型，使其具备远超对话场景的丰富动作先验知识。接着，我们提出了一种音频感知的规划-再填充架构，将句子级语义规划与帧级韵律驱动的插值解耦，从而使生成的动作既在语义上恰当，又在节奏上与语音对齐。实验表明，SentiAvatar在SuSuInterActs（R@1 43.64%，接近最佳基线的2倍）和BEATv2（FGD 4.941，BC 8.078）上均达到最先进水平，仅需0.3秒即可生成6秒的输出，并支持无限多轮流式生成。源代码、模型和数据集可通过此https URL获取。
URL
https://arxiv.org/abs/2604.02908
