Abstract
Recent work has shown that, in generative modeling, cross-entropy loss improves smoothly with model size and training compute, following a power law plus constant scaling law. One challenge in extending these results to reinforcement learning is that the main performance objective of interest, mean episode return, need not vary smoothly. To overcome this, we introduce *intrinsic performance*, a monotonic function of the return defined as the minimum compute required to achieve the given return across a family of models of different sizes. We find that, across a range of environments, intrinsic performance scales as a power law in model size and environment interactions. Consequently, as in generative modeling, the optimal model size scales as a power law in the training compute budget. Furthermore, we study how this relationship varies with the environment and with other properties of the training setup. In particular, using a toy MNIST-based environment, we show that varying the "horizon length" of the task mostly changes the coefficient but not the exponent of this relationship.
Abstract (translated)
最近的研究表明,在生成建模中,交叉熵损失随着模型大小和训练计算的改善而平滑地增加,遵循一个幂函数加常数指数级增长的规律。将这些结果扩展到奖励学习中的一个挑战是,感兴趣的主要性能目标,即平均事件回报,并不一定随着模型大小和环境因素的变化而平滑变化。为了克服这个问题,我们引入了 *内在性能*,它是一个单调函数,定义为在一组不同模型大小下实现给定回报所需的最小计算量。我们发现,在多种环境中,内在性能随着模型大小和环境交互而呈现出幂函数分布。因此,与生成建模类似,最优模型大小在训练计算预算中呈现出幂函数分布。此外,我们研究了这个关系与环境因素和训练设置其他特性之间的关系。特别是,使用一个基于手写数字识别玩具环境,我们表明,Task的“截止日期长度”的变化主要影响了系数,而不是这个关系的指数级。
URL
https://arxiv.org/abs/2301.13442
