Abstract
Most state-of-the-art approaches for weather and climate modeling are based on physics-informed numerical models of the atmosphere. These approaches aim to model the non-linear dynamics and complex interactions between multiple variables, which are challenging to approximate. Additionally, many such numerical models are computationally intensive, especially when modeling the atmospheric phenomenon at a fine-grained spatial and temporal resolution. Recent data-driven approaches based on machine learning instead aim to directly solve a downstream forecasting or projection task by learning a data-driven functional mapping using deep neural networks. However, these networks are trained using curated and homogeneous climate datasets for specific spatiotemporal tasks, and thus lack the generality of numerical models. We develop and demonstrate ClimaX, a flexible and generalizable deep learning model for weather and climate science that can be trained using heterogeneous datasets spanning different variables, spatio-temporal coverage, and physical groundings. ClimaX extends the Transformer architecture with novel encoding and aggregation blocks that allow effective use of available compute while maintaining general utility. ClimaX is pre-trained with a self-supervised learning objective on climate datasets derived from CMIP6. The pre-trained ClimaX can then be fine-tuned to address a breadth of climate and weather tasks, including those that involve atmospheric variables and spatio-temporal scales unseen during pretraining. Compared to existing data-driven baselines, we show that this generality in ClimaX results in superior performance on benchmarks for weather forecasting and climate projections, even when pretrained at lower resolutions and compute budgets.
Abstract (translated)
大多数的天气和气候建模先进方法是基于物理 informed 的大气数值模型。这些方法旨在模拟非线性动态和多个变量之间的复杂相互作用,很难精确模拟。此外,许多这样的数值模型的计算量很大,特别是在精细的空间和时间分辨率下模拟大气现象。最近,基于机器学习的数据驱动方法旨在直接解决下游预测或投影任务,通过使用深度神经网络学习数据驱动的功能映射。但是,这些网络使用 curated 和相同的气候数据集针对特定的时间和空间任务进行训练,因此缺乏数值模型的一般性。我们开发和展示了 ClimaX,这是一款天气和气候科学中的灵活和通用深度学习模型,可以使用不同变量、时间和空间覆盖面的数据进行训练。ClimaX 扩展了变分自编码器和聚合块的新架构,使其能够有效利用可用的计算,同时保持通用性。ClimaX 在从 CMIP6 中提取的气候数据集上进行了自我监督学习目标的训练。训练后的 ClimaX 可以微调以解决广泛的气候和天气任务,包括在训练期间未观察到的大气变量和时间和空间尺度的任务。与现有的数据驱动基准相比,我们表明,ClimaX 的一般性导致在天气预报和气候预测基准上的卓越性能,即使在较低的分辨率和计算预算下进行训练。
URL
https://arxiv.org/abs/2301.10343
