Abstract
Multimodal generative models that can understand and generate across multiple modalities are dominated by autoregressive (AR) approaches, which process tokens sequentially from left to right, or top to bottom. These models jointly handle images, text, video, and audio for various tasks such as image captioning, question answering, and image generation. In this work, we explore discrete diffusion models as a unified generative formulation in the joint text and image domain, building upon their recent success in text generation. Discrete diffusion models offer several advantages over AR models, including improved control over quality versus diversity of generated samples, the ability to perform joint multimodal inpainting (across both text and image domains), and greater controllability in generation through guidance. Leveraging these benefits, we present the first Unified Multimodal Discrete Diffusion (UniDisc) model which is capable of jointly understanding and generating text and images for a variety of downstream tasks. We compare UniDisc to multimodal AR models, performing a scaling analysis and demonstrating that UniDisc outperforms them in terms of both performance and inference-time compute, enhanced controllability, editability, inpainting, and flexible trade-off between inference time and generation quality. Code and additional visualizations are available at this https URL.
Abstract (translated)
多模态生成模型能够理解和生成跨多种模式（如图像、文本、视频和音频）的数据，在任务如图像描述、问答和图像生成中发挥作用。这些模型主要采用自回归（AR）方法，按从左到右或从上到下的顺序处理标记。在这项工作中，我们探讨了离散扩散模型作为联合文本和图像领域的统一生成框架，并在此前成功应用于文本生成的基础上进行研究。离散扩散模型相比自回归模型具有若干优势，包括对生成样本质量和多样性的更好控制、能够在文本和图像域内执行联合多模态插补以及通过指导实现更可控的生成能力。 利用这些优点，我们提出了首个统一多模态离散扩散（UniDisc）模型，该模型能够同时理解和生成用于各种下游任务的文本和图像。我们将UniDisc与多模态自回归模型进行比较，并进行了规模分析，证明在性能、推理时间计算、增强可控性、可编辑性和插补以及在推理时间和生成质量之间的灵活权衡方面，UniDisc优于它们。 代码及相关可视化内容可在以下链接获得：[请在此处插入实际URL]。
URL
https://arxiv.org/abs/2503.20853
