Abstract
Hands are central to interacting with our surroundings and conveying gestures, making their inclusion essential for full-body motion synthesis. Despite this, existing human motion synthesis methods fall short: some ignore hand motions entirely, while others generate full-body motions only for narrowly scoped tasks under highly constrained settings. A key obstacle is the lack of large-scale datasets that jointly capture diverse full-body motion with detailed hand articulation. While some datasets capture both, they are limited in scale and diversity. Conversely, large-scale datasets typically focus either on body motion without hands or on hand motions without the body. To overcome this, we curate and unify existing hand motion datasets with large-scale body motion data to generate full-body sequences that capture both hand and body. We then propose the first diffusion-based unconditional full-body motion prior, FUSION, which jointly models body and hand motion. Despite using a pose-based motion representation, FUSION surpasses state-of-the-art skeletal control models on the Keypoint Tracking task in the HumanML3D dataset and achieves superior motion naturalness. Beyond standard benchmarks, we demonstrate that FUSION can go beyond typical uses of motion priors through two applications: (1) generating detailed full-body motion including fingers during interaction given the motion of an object, and (2) generating Self-Interaction motions using an LLM to transform natural language cues into actionable motion constraints. For these applications, we develop an optimization pipeline that refines the latent space of our diffusion model to generate task-specific motions. Experiments on these tasks highlight precise control over hand motion while maintaining plausible full-body coordination. The code will be public.
Abstract (translated)
手部在与周围环境互动以及传达手势方面起着关键作用，因此在全身动作合成中包括手部运动是必不可少的。然而，现有的人体动作合成方法存在不足：一些方法完全忽略了手部动作，而另一些则仅能在高度受限的情况下生成窄范围任务的全身体动。一个主要障碍在于缺乏大规模的数据集，这些数据集能够同时捕捉多样化的全身动作和详细的手部关节运动。虽然有些数据集能捕获两者，但它们在规模和多样性方面存在限制。相比之下，大型数据集通常要么专注于没有手的身体运动，要么专注于没有身体的手部运动。 为了解决这个问题，我们整合并统一现有的手部运动数据集与大规模身体运动数据集，以生成同时捕捉手部和身体动作的全身序列。随后，我们提出了首个基于扩散模型的无条件全身体动先验FUSION（Full-body UNconditional motion SynthesIzation），该模型能够联合建模身体和手部的动作。尽管使用的是基于姿势的动作表示方式，但FUSION在HumanML3D数据集的关键点跟踪任务上超越了最新的骨骼控制模型，并实现了更自然的运动效果。 除了标准基准测试外，我们通过两个应用展示了FUSION可以超越传统动作用途：（1）根据物体的移动生成详细的全身动作，包括手指互动；（2）利用LLM将自然语言提示转化为可操作的动作约束来生成自我交互动作。为了这些应用程序，我们开发了一个优化管道，该管道能够细化我们的扩散模型的潜在空间以生成特定任务的运动。在这些任务上的实验突显了对手部动作的精确控制同时保持合理的全身协调性。代码将公开提供。
URL
https://arxiv.org/abs/2601.03959
