The Llama 3 Herd of Models

• Abstract
• Abstract (translated)
• URL
• PDF

Abstract

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

Abstract (translated)

现代人工智能（AI）系统是由基础模型驱动的。本文介绍了一种名为Llama 3的新基础模型。它是一群支持多语言性、编码、推理和工具使用的语言模型的集合。我们最大的模型是一个具有405B参数的密集Transformer，具有最大的上下文窗口为128K个标记。本文对Llama 3进行了广泛的实证评估。我们发现，Llama 3在各种任务上的质量与诸如GPT-4等领先语言模型相当。我们公开发布了Llama 3，包括预训练和后训练的405B参数语言模型以及Llama Guard 3模型，用于输入和输出安全。本文还描述了我们将图像、视频和语音功能通过组合方法集成到Llama 3中的实验结果。我们观察到，这种方法在图像、视频和语音识别任务上与最先进的技术具有竞争力。然而，这些模型尚未被广泛发布，因为它们仍在开发中。

URL

https://arxiv.org/abs/2407.21783

PDF Copy

https://arxiv.org/pdf/2407.21783.pdf