Abstract
Lowering the precision of neural networks from the prevalent 32-bit precision has long been considered harmful to performance, despite the gain in space and time. Many works propose various techniques to implement half-precision neural networks, but none study pure 16-bit settings. This paper investigates the unexpected performance gain of pure 16-bit neural networks over the 32-bit networks in classification tasks. We present extensive experimental results that favorably compare various 16-bit neural networks' performance to those of the 32-bit models. In addition, a theoretical analysis of the efficiency of 16-bit models is provided, which is coupled with empirical evidence to back it up. Finally, we discuss situations in which low-precision training is indeed detrimental.
Abstract (translated)
降低神经网络的精度一直被认为是对性能有害的,尽管在时间和空间上都有所进步。许多工作提出了各种方法来实施半精度神经网络,但都没有研究纯粹的16位设置。本文研究了纯16位神经网络在分类任务中的 unexpected 性能 gain,并呈现了广泛的实验结果,它们与32位模型的性能进行积极比较。此外,提供了16位模型的效率理论分析,并结合经验证据支持它。最后,我们讨论了低精度训练确实有害的情况。
URL
https://arxiv.org/abs/2301.12809