Abstract
We give a comprehensive analysis of transformers as time series foundation models, focusing on their approximation and generalization capabilities. First, we demonstrate that there exist transformers that fit an autoregressive model on input univariate time series via gradient descent. We then analyze MOIRAI, a multivariate time series foundation model capable of handling an arbitrary number of covariates. We prove that it is capable of automatically fitting autoregressive models with an arbitrary number of covariates, offering insights into its design and empirical success. For generalization, we establish bounds for pretraining when the data satisfies Dobrushin's condition. Experiments support our theoretical findings, highlighting the efficacy of transformers as time series foundation models.
Abstract (translated)
我们对变压器作为时间序列基础模型进行了全面分析，重点关注它们的近似和泛化能力。首先，我们证明存在可以通过梯度下降在输入的一维时间序列上拟合自回归模型的变压器。然后，我们研究了MOIRAI，这是一种可以处理任意数量协变量的多变量时间序列基础模型，并证明它可以自动适应具有任意数量协变量的自回归模型，从而提供对其设计和实证成功的见解。对于泛化能力，我们在数据满足Dobrushin条件的情况下建立了预训练的界限。实验结果支持了我们的理论发现，突显了变压器作为时间序列基础模型的有效性。
URL
https://arxiv.org/abs/2502.03383
