Abstract
In this work, we propose Dimple, the first Discrete Diffusion Multimodal Large Language Model (DMLLM). We observe that training with a purely discrete diffusion approach leads to significant training instability, suboptimal performance, and severe length bias issues. To address these challenges, we design a novel training paradigm that combines an initial autoregressive phase with a subsequent diffusion phase. This approach yields the Dimple-7B model, trained on the same dataset and using a similar training pipeline as LLaVA-NEXT. Dimple-7B ultimately surpasses LLaVA-NEXT in performance by 3.9%, demonstrating that DMLLM can achieve performance comparable to that of autoregressive models. To improve inference efficiency, we propose a decoding strategy termed confident decoding, which dynamically adjusts the number of tokens generated at each step, significantly reducing the number of generation iterations. In autoregressive models, the number of forward iterations during generation equals the response length. With confident decoding, however, the number of iterations needed by Dimple is even only $\frac{\text{response length}}{3}$. We also re-implement the prefilling technique in autoregressive models and demonstrate that it does not significantly impact performance on most benchmark evaluations, while offering a speedup of 1.5x to 7x. Additionally, we explore Dimple's capability to precisely control its response using structure priors. These priors enable structured responses in a manner distinct from instruction-based or chain-of-thought prompting, and allow fine-grained control over response format and length, which is difficult to achieve in autoregressive models. Overall, this work validates the feasibility and advantages of DMLLM and enhances its inference efficiency and controllability. Code and models are available at this https URL.
Abstract (translated)
在这项工作中，我们提出了Dimple，这是首个离散扩散多模态大型语言模型（DMLLM）。我们观察到，使用纯粹的离散扩散方法进行训练会导致显著的训练不稳定、次优性能和严重的长度偏差问题。为了应对这些挑战，我们设计了一种新的训练范式，该范式结合了初始自回归阶段与后续的扩散阶段。这种方法生成了Dimple-7B模型，其在相同的语料库上进行了训练，并使用了类似于LLaVA-NEXT的训练管道。最终，Dimple-7B以3.9%的优势超越了LLaVA-NEXT，这表明DMLLM可以实现与自回归模型相当的性能。 为了提高推理效率，我们提出了一种名为“自信解码”的解码策略，该策略在每个步骤中动态调整生成令牌的数量，显著减少了生成迭代次数。在自回归模型中，生成期间的前向迭代次数等于响应长度。然而，在使用自信解码时，Dimple所需的迭代次数仅为响应长度的$\frac{1}{3}$。 此外，我们重新实现了自回归模型中的填充技术，并展示了这种技术对大多数基准评估性能影响不大，但提供了1.5倍到7倍的速度提升。我们也探讨了Dimple利用结构先验精准控制其响应的能力。这些先验使得以不同于指令或链式思考提示的方式生成结构化回复成为可能，从而可以精确地控制回复格式和长度，而这在自回归模型中是难以实现的。 总的来说，这项工作验证了DMLLM的可行性和优势，并提高了它的推理效率和可控性。代码与模型可在[此处](https://this-url.com)获取。
URL
https://arxiv.org/abs/2505.16990
