Abstract
Many real-world datasets are time series that are sequentially collected and contain rich temporal information. Thus, a common interest in practice is to capture dynamics of time series and predict their future evolutions. To this end, the recurrent neural network (RNN) has been a prevalent and effective machine learning option, which admits a nonlinear state-space model representation. Motivated by the resemblance between RNN and Kalman filter (KF) for linear state-space models, we propose in this paper Innovation-driven RNN (IRNN), a novel RNN architecture tailored to time-series data modeling and prediction tasks. By adapting the concept of "innovation" from KF to RNN, past prediction errors are adopted as additional input signals to update hidden states of RNN and boost prediction performance. Since innovation data depend on network parameters, existing training algorithms for RNN do not apply to IRNN straightforwardly. Thus, a tailored training algorithm dubbed input updating-based back-propagation through time (IU-BPTT) is further proposed, which alternates between updating innovations and optimizing network parameters via gradient descent. Experiments on real-world benchmark datasets show that the integration of innovations into various forms of RNN leads to remarkably improved prediction accuracy of IRNN without increasing the training cost substantially.
Abstract (translated)
许多现实世界的数据集都是按时间顺序收集的时间序列数据，包含丰富的时序信息。因此，在实践中一个共同的兴趣是捕捉时间序列的动力学特性并预测它们的未来演变趋势。为此，递归神经网络（RNN）已成为一种普遍且有效的机器学习选项，它可以表示非线性状态空间模型。受RNN与线性状态空间模型中的卡尔曼滤波器（KF）之间的相似性的启发，本文提出了创新驱动式RNN（IRNN），这是一种新型的针对时间序列数据建模和预测任务定制化的RNN架构。通过将“创新”这一概念从KF引入到RNN中，我们采用了过去预测误差作为额外输入信号来更新RNN的状态，并提高了预测性能。由于创新数据依赖于网络参数，现有的适用于RNN的训练算法不能直接应用于IRNN。因此，进一步提出了一种定制化的训练算法——基于输入更新的反向传播通过时间（IU-BPTT），该算法交替进行创新更新和利用梯度下降优化网络参数。在现实世界的基准数据集上的实验表明，在不显著增加训练成本的情况下，将创新整合到各种形式的RNN中可以大大提高IRNN的预测准确性。
URL
https://arxiv.org/abs/2505.05916
