Abstract
The volume of unlabelled Earth observation (EO) data is huge, but many important applications lack labelled training data. However, EO data offers the unique opportunity to pair data from different modalities and sensors automatically based on geographic location and time, at virtually no human labor cost. We seize this opportunity to create a diverse multi-modal pretraining dataset at global scale. Using this new corpus of 1.2 million locations, we propose a Multi-Pretext Masked Autoencoder (MP-MAE) approach to learn general-purpose representations for optical satellite images. Our approach builds on the ConvNeXt V2 architecture, a fully convolutional masked autoencoder (MAE). Drawing upon a suite of multi-modal pretext tasks, we demonstrate that our MP-MAE approach outperforms both MAEs pretrained on ImageNet and MAEs pretrained on domain-specific satellite images. This is shown on several downstream tasks including image classification and semantic segmentation. We find that multi-modal pretraining notably improves the linear probing performance, e.g. 4pp on BigEarthNet and 16pp on So2Sat, compared to pretraining on optical satellite images only. We show that this also leads to better label and parameter efficiency which are crucial aspects in global scale applications.
Abstract (translated)
未标记的地球观测（EO）数据的体积巨大，但许多重要的应用缺乏标记的训练数据。然而，EO数据为根据地理位置和时间自动对不同模式和传感器数据进行对齐提供了独特的机会，几乎不需要任何人类劳动成本。我们抓住了这个机会，在全球范围内创建了一个多样化的多模态预训练数据集。利用这个包含1200万个位置的新数据集，我们提出了一个多预文本掩码自编码器（MP-MAE）方法，用于学习光学卫星图像的通用表示。我们的方法基于ConvNeXt V2架构，这是一个完全卷积掩码自编码器（MAE）。利用一系列多模态预文本任务，我们证明了我们的MP-MAE方法在ImageNet上预训练的MAEs和预训练在领域特定卫星图像上的MAEs之间都表现优异。这在几个下游任务中得到了证实，包括图像分类和语义分割。我们发现，多模态预训练显著提高了线性探测性能，例如在BigEarthNet上的4pp和So2Sat上的16pp，而仅预训练在光学卫星图像上。我们还证明了这还导致了更好的标签和参数效率，这是在全局应用中至关重要的一些方面。
URL
https://arxiv.org/abs/2405.02771
