Abstract
Various recent experimental results show that large language models (LLM) exhibit emergent abilities that are not present in small models. System performance is greatly improved after passing a certain critical threshold of scale. In this letter, we provide a simple explanation for such a phase transition phenomenon. For this, we model an LLM as a sequence-to-sequence random function. Instead of using instant generation at each step, we use a list decoder that keeps a list of candidate sequences at each step and defers the generation of the output sequence at the end. We show that there is a critical threshold such that the expected number of erroneous candidate sequences remains bounded when an LLM is below the threshold, and it grows exponentially when an LLM is above the threshold. Such a threshold is related to the basic reproduction number in a contagious disease.
Abstract (translated)
各种最近的实验结果表明，大型语言模型(LLM)表现出小型模型无法出现的 emergent 能力。系统性能在达到某个规模 critical 阈值后极大地改善。在本信中，我们提供了这种相位转移现象的简单解释。为此，我们将 LLM 建模为序列到序列随机函数。 Instead of 使用每个步骤的即时生成，我们使用一个列表解码器，在每个步骤中保持一个列表，并在最后推迟生成输出序列的生成。我们表明，存在一个 critical 阈值，即当 LLM 低于该阈值时，错误候选序列的预计数量将保持有限值，而当 LLM 高于该阈值时，它会呈指数级增长。这种阈值与传染病的基本复制数有关。
URL
https://arxiv.org/abs/2303.13112
