Abstract
We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA fine-tuning. The entire training cycle is super efficient, which takes 8 hours on one 8xA800 (80G) GPU machine. The resulted model exhibits superior performances across a broad range of evaluation tasks, such as NIHS, topic retrieval, and long-context language understanding; meanwhile, it also well preserves the original capability over short contexts. The dramatic context extension is mainly attributed to merely 3.5K synthetic training samples generated by GPT-4 , which indicates the LLMs' inherent (yet largely underestimated) potential to extend its original context length. In fact, the context length could be extended far beyond 80K with more computation resources. Therefore, the team will publicly release the entire resources (including data, model, data generation pipeline, training code) so as to facilitate the future research from the community: \url{this https URL}.
Abstract (translated)
我们通过 QLoRA 微调将Llama-3-8B-Instruct的上下文长度从8K扩展到80K。整个训练周期非常高效，只需在单个8xA800（80G）GPU机器上花费8小时。所得到的模型在广泛的评估任务中都表现出卓越的性能，例如NIHS、主题检索和长文本理解；同时，它还能够在短上下文中很好地保留原始功能。戏剧性的上下文扩展主要归功于由GPT-4生成的3.5K个合成训练样本，这表明LLM固有的（然而却被低估）延展原始上下文长度的潜力。实际上，使用更多的计算资源，上下文长度可以大大扩展。因此，团队将公开发布整个资源（包括数据、模型、数据生成管道和训练代码），以便于未来社区的研究：\url{this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <https://this <
URL
https://arxiv.org/abs/2404.19553
