Abstract
We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the model. Third, we present a novel self-speculative decoding solution where we exit at early layers and verify and correct with remaining layers of the model. Our proposed self-speculative decoding approach has less memory footprint than other speculative decoding approaches and benefits from shared compute and activations of the draft and verification stages. We run experiments on different Llama model sizes on different types of training: pretraining from scratch, continual pretraining, finetuning on specific data domain, and finetuning on specific task. We implement our inference solution and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task.
Abstract (translated)
我们提出了LayerSkip，这是一种加速大型语言模型（LLM）推理速度的端到端解决方案。首先，在训练过程中，我们应用层下落，对于较早的层，下落率较低，对于较晚的层，下落率较高，并且有一个早期的退出损失，其中所有Transformer层都共享相同的退出。其次，在推理过程中，我们证明了这种训练方法在较早的层上增加了早期退出模型的准确性，而没有添加任何辅助层或模块到模型中。第三，我们提出了一个新颖的自适应解码解决方案，其中我们在较早的层退出，并使用模型的剩余层来验证和纠正。我们针对LLama模型的大小和不同类型的训练进行了实验：从头预训练，继续预训练，针对特定数据领域的微调，以及针对特定任务的微调。我们实现了我们的推理解决方案，并在综述的CNN/DM文档上展示了速度提高至2.16倍，在编码上提高了1.82倍，在TOPv2语义解析任务上提高了2.0倍。
URL
https://arxiv.org/abs/2404.16710
