Abstract
This paper revisits the standard pretrain-then-finetune paradigm used in computer vision for visual recognition tasks. Typically, state-of-the-art foundation models are pretrained using large scale (weakly) supervised datasets with billions of images. We introduce an additional pre-pretraining stage that is simple and uses the self-supervised MAE technique to initialize the model. While MAE has only been shown to scale with the size of models, we find that it scales with the size of the training dataset as well. Thus, our MAE-based pre-pretraining scales with both model and data size making it applicable for training foundation models. Pre-pretraining consistently improves both the model convergence and the downstream transfer performance across a range of model scales (millions to billions of parameters), and dataset sizes (millions to billions of images). We measure the effectiveness of pre-pretraining on 10 different visual recognition tasks spanning image classification, video recognition, object detection, low-shot classification and zero-shot recognition. Our largest model achieves new state-of-the-art results on iNaturalist-18 (91.3%), 1-shot ImageNet-1k (62.1%), and zero-shot transfer on Food-101 (96.0%). Our study reveals that model initialization plays a significant role, even for web-scale pretraining with billions of images.
Abstract (translated)
这篇文章重写了计算机视觉中用于视觉识别任务的标准预训练-再微调范式。通常，最先进的基础模型是通过大规模(较弱)监督数据集训练的，包含数百万图像。我们引入了一个简单的预训练阶段，并使用自监督MAE技术初始化模型。虽然MAE只表现出与模型大小的关系，但我们发现它与训练数据集大小也有关系。因此，我们的MAE基于预训练方法适用于训练基础模型。预训练 consistently 改善模型收敛和下游转移性能，涵盖了模型大小(数百万到数十亿参数)和数据大小(数百万到数十亿图像)。我们测试了10个不同的视觉识别任务，包括图像分类、视频识别、对象检测、低尺度分类和零尺度识别。我们最大的模型在iNaturalist-18上取得了新的最先进的结果(91.3%)，在1-视角的ImageNet-1k上取得了62.1%的准确率，并在Food-101上实现了零视角转移(96.0%)。我们的研究表明，模型初始化在包含数十亿图像的大规模预训练任务中发挥着重要作用。
URL
https://arxiv.org/abs/2303.13496
