Abstract
Online continual learning (OCL) methods adapt to changing environments without forgetting past knowledge. Similarly, online time series forecasting (OTSF) is a real-world problem where data evolve in time and success depends on both rapid adaptation and long-term memory. Indeed, time-varying and regime-switching forecasting models have been extensively studied, offering a strong justification for the use of OCL in these settings. Building on recent work that applies OCL to OTSF, this paper aims to strengthen the theoretical and practical connections between time series methods and OCL. First, we reframe neural network optimization as a parameter filtering problem, showing that natural gradient descent is a score-driven method and proving its information-theoretic optimality. Then, we show that using a Student's t likelihood in addition to natural gradient induces a bounded update, which improves robustness to outliers. Finally, we introduce Natural Score-driven Replay (NatSR), which combines our robust optimizer with a replay buffer and a dynamic scale heuristic that improves fast adaptation at regime drifts. Empirical results demonstrate that NatSR achieves stronger forecasting performance than more complex state-of-the-art methods.
Abstract (translated)
在线连续学习（OCL）方法能够在不忘记过去知识的情况下适应环境的变化。同样，实时序列预测（OTSF）是现实世界中的一个问题，在这个问题中数据会随时间演变，成功取决于快速适应和长期记忆的能力。实际上，时变和制度转换的预测模型已经得到了广泛的研究，这为在这些场景下使用OCL提供了强大的理论依据。基于最近将OCL应用于OTSF的工作，本文旨在加强时间序列方法与OCL之间的理论和实践联系。 首先，我们将神经网络优化重新定义为参数过滤问题，并展示了自然梯度下降是一种评分驱动的方法，并证明了其信息论上的最优性。然后，我们表明，在使用自然梯度的同时采用Student's t似然函数可以实现有界更新，从而提高对离群值的鲁棒性。最后，我们提出了自然评分驱动重放（NatSR），该方法结合了我们的鲁棒优化器、回放缓存以及动态尺度启发式策略，以在制度漂移期间改善快速适应能力。 实验证据表明，NatSR在预测性能方面优于更复杂的最新方法。
URL
https://arxiv.org/abs/2601.12931
