Abstract
Inference with Multimodal Large Language Models (MLLMs) is slow due to their large-language-model backbone which suffers from memory bandwidth bottleneck and generates tokens auto-regressively. In this paper, we explore the application of speculative decoding to enhance the inference efficiency of MLLMs, specifically the LLaVA 7B model. We show that a language-only model can serve as a good draft model for speculative decoding with LLaVA 7B, bypassing the need for image tokens and their associated processing components from the draft model. Our experiments across three different tasks show that speculative decoding can achieve a memory-bound speedup of up to 2.37$\times$ using a 115M parameter language model that we trained from scratch. Additionally, we introduce a compact LLaVA draft model incorporating an image adapter, which shows marginal performance gains in image captioning while maintaining comparable results in other tasks.
Abstract (translated)
使用多模态大型语言模型（MLLMs）进行推理速度较慢，因为它们的大型语言模型骨架存在内存带宽瓶颈并生成逐个生成的标记。在本文中，我们探讨了使用类推断来提高MLLMs的推理效率，特别是LLaVa 7B模型。我们证明了仅使用语言模型的类推断可以作为良好的原型模型，绕过原型模型中图像标记和相关的处理组件的需求。我们在三个不同的任务上的实验表明，使用我们从头训练的115M参数语言模型可以实现多达2.37倍的内存加速。此外，我们还介绍了一个紧凑的LLaVa原型模型，其中包含图像适配器，在保持与其它任务类似的结果的同时，在图像描述性任务上表现出微小的性能提升。
URL
https://arxiv.org/abs/2404.08856
