Abstract
We argue that diffusion models' success in modeling complex distributions is, for the most part, coming from their input conditioning. This paper investigates the representation used to condition diffusion models from the perspective that ideal representations should improve sample fidelity, be easy to generate, and be compositional to allow out-of-training samples generation. We introduce Discrete Latent Code (DLC), an image representation derived from Simplicial Embeddings trained with a self-supervised learning objective. DLCs are sequences of discrete tokens, as opposed to the standard continuous image embeddings. They are easy to generate and their compositionality enables sampling of novel images beyond the training distribution. Diffusion models trained with DLCs have improved generation fidelity, establishing a new state-of-the-art for unconditional image generation on ImageNet. Additionally, we show that composing DLCs allows the image generator to produce out-of-distribution samples that coherently combine the semantics of images in diverse ways. Finally, we showcase how DLCs can enable text-to-image generation by leveraging large-scale pretrained language models. We efficiently finetune a text diffusion language model to generate DLCs that produce novel samples outside of the image generator training distribution.
Abstract (translated)
我们认为，扩散模型在建模复杂分布方面取得的成功，在很大程度上源于它们的输入条件设置。本文从理想表示应提高样本保真度、易于生成且具有组合性以支持训练数据之外的新样本生成这一视角出发，研究了用于调节扩散模型的表示方法。我们引入了离散潜在码（Discrete Latent Code, DLC），这是一种源自单纯形嵌入并在自监督学习目标下进行训练的图像表示形式。DLC是一系列离散令牌序列，而不是标准的连续图像嵌入。它们易于生成，并且其组合性使得能够合成超出训练数据分布的新颖图像。使用DLC训练的扩散模型在无条件图像生成方面提高了生成保真度，在ImageNet数据集上建立了新的最先进的性能。此外，我们展示了通过组合DLC可以促使图像生成器产生出训练分布之外、并且能连贯地结合不同图像语义的新样本。最后，我们演示了如何利用大规模的预训练语言模型来实现文本到图像的生成。我们将一个文本扩散语言模型进行有效微调，使其能够生成超出图像生成器训练数据分布范围的新颖DLC样本。
URL
https://arxiv.org/abs/2507.12318
