Abstract
Semantic-driven 3D shape generation aims to generate 3D objects conditioned on text. Previous works face problems with single-category generation, low-frequency 3D details, and requiring a large number of paired datasets for training. To tackle these challenges, we propose a multi-category conditional diffusion model. Specifically, 1) to alleviate the problem of lack of large-scale paired data, we bridge the text, 2D image and 3D shape based on the pre-trained CLIP model, and 2) to obtain the multi-category 3D shape feature, we apply the conditional flow model to generate 3D shape vector conditioned on CLIP embedding. 3) to generate multi-category 3D shape, we employ the hidden-layer diffusion model conditioned on the multi-category shape vector, which greatly reduces the training time and memory consumption.
Abstract (translated)
语义驱动的三维形状生成旨在根据文本生成三维对象。以前的工作面临单类别生成、低频率的三维细节和需要大量配对数据训练的问题。为了解决这些问题,我们提出了一种多类别条件扩散模型。具体来说,1) 为了减轻缺乏大规模配对数据的问题,我们基于预先训练的CLIP模型将文本、2D图像和3D形状连接起来,2) 为了获取多类别3D形状特征,我们应用条件流模型,根据CLIP嵌入生成3D形状向量。3) 为了生成多类别3D形状,我们采用基于多类别形状向量的隐藏层扩散模型,这大大缩短了训练时间和内存消耗。
URL
https://arxiv.org/abs/2301.13591
