Abstract
We present a novel method to generate human motion to populate 3D indoor scenes. It can be controlled with various combinations of conditioning signals such as a path in a scene, target poses, past motions, and scenes represented as 3D point clouds. State-of-the-art methods are either models specialized to one single setting, require vast amounts of high-quality and diverse training data, or are unconditional models that do not integrate scene or other contextual information. As a consequence, they have limited applicability and rely on costly training data. To address these limitations, we propose a new method ,dubbed Purposer, based on neural discrete representation learning. Our model is capable of exploiting, in a flexible manner, different types of information already present in open access large-scale datasets such as AMASS. First, we encode unconditional human motion into a discrete latent space. Second, an autoregressive generative model, conditioned with key contextual information, either with prompting or additive tokens, and trained for next-step prediction in this space, synthesizes sequences of latent indices. We further design a novel conditioning block to handle future conditioning information in such a causal model by using a network with two branches to compute separate stacks of features. In this manner, Purposer can generate realistic motion sequences in diverse test scenes. Through exhaustive evaluation, we demonstrate that our multi-contextual solution outperforms existing specialized approaches for specific contextual information, both in terms of quality and diversity. Our model is trained with short sequences, but a byproduct of being able to use various conditioning signals is that at test time different combinations can be used to chain short sequences together and generate long motions within a context scene.
Abstract (translated)
我们提出了一种名为Purposer的新方法，基于神经离散表示学习。我们的模型能够以灵活的方式利用开放访问的大规模数据集AMASS中已经存在的不同类型的信息。首先，我们将无条件的人类运动编码到一个离散的潜在空间中。然后，一个条件生成模型，通过关键的上下文信息条件，以提示或添加标记的方式进行训练，并在该空间中进行下一步预测，合成了一系列的潜在索引。我们进一步设计了一个新的条件模块，用于在具有因果关系的模型中处理未来的条件信息，通过使用具有两个分支的网络计算不同的特征栈。这样，Purposer可以在各种测试场景中生成逼真的运动序列。通过彻底的评估，我们证明了我们的多上下文解决方案在特定上下文信息方面的现有专业方法中具有优越性，无论是质量还是多样性。我们的模型使用短序列进行训练，但能够使用各种上下文信号的原因是，在测试时可以使用不同的组合将短序列串联起来并在上下文场景中生成长动作。
URL
https://arxiv.org/abs/2404.12942
