Abstract
Hoffmann et al. (2022) propose three methods for estimating a compute-optimal scaling law. We attempt to replicate their third estimation procedure, which involves fitting a parametric loss function to a reconstruction of data from their plots. We find that the reported estimates are inconsistent with their first two estimation methods, fail at fitting the extracted data, and report implausibly narrow confidence intervals--intervals this narrow would require over 600,000 experiments, while they likely only ran fewer than 500. In contrast, our rederivation of the scaling law using the third approach yields results that are compatible with the findings from the first two estimation procedures described by Hoffmann et al.
Abstract (translated)
霍夫曼等人（2022）提出了三种估计计算最优缩放定律的方法。我们尝试复制他们的第三种估计方法，该方法涉及将参数损失函数拟合到从他们的图上提取的数据的重构上。我们发现，报道的估计值与他们前两种估计方法不一致，在拟合提取的数据时失败，并且报告了不合逻辑的置信区间——即使缩小得如此之窄，也需要超过600,000个实验，而他们很可能只运行了不到500个实验。相比之下，我们使用第三种方法重新推导缩放定律，得到的结果与霍夫曼等人（2022）描述的第一和第二种估计方法得出的结果相一致。
URL
https://arxiv.org/abs/2404.10102
