Abstract
Time Series foundation models (TSFMs) deliver strong forecasting performance through large-scale pretraining, but their large parameter sizes make deployment costly. While knowledge distillation offers a natural and effective approach for model compression, techniques developed for general machine learning tasks are not directly applicable to time series forecasting due to the unique characteristics. To address this, we present DistilTS, the first distillation framework specifically designed for TSFMs. DistilTS addresses two key challenges: (1) task difficulty discrepancy, specific to forecasting, where uniform weighting makes optimization dominated by easier short-term horizons, while long-term horizons receive weaker supervision; and (2) architecture discrepancy, a general challenge in distillation, for which we design an alignment mechanism in the time series forecasting. To overcome these issues, DistilTS introduces horizon-weighted objectives to balance learning across horizons, and a temporal alignment strategy that reduces architectural mismatch, enabling compact models. Experiments on multiple benchmarks demonstrate that DistilTS achieves forecasting performance comparable to full-sized TSFMs, while reducing parameters by up to 1/150 and accelerating inference by up to 6000x. Code is available at: this https URL.
Abstract (translated)
时间序列基础模型（TSFM）通过大规模预训练提供了强大的预测性能，但其庞大的参数规模使得部署成本高昂。虽然知识蒸馏为模型压缩提供了一种自然且有效的方法，但由于时间序列预测的独特特性，通用机器学习任务中开发的技术并不直接适用。为此，我们提出了DistilTS，这是第一个专门针对TSFM设计的知识蒸馏框架。 DistilTS解决了两个关键挑战：（1）预测特有的任务难度差异问题，在这种情况下，均匀加权使得优化主要由较容易的短期地平线主导，而长期地平线则受到更弱的监督；（2）架构差异，这是知识蒸馏中的一般性挑战，为此我们在时间序列预测中设计了一种对齐机制。为克服这些问题，DistilTS引入了基于地平线加权的目标函数以平衡不同地平线之间的学习，并采用一种减少架构不匹配的时间对齐策略，从而实现模型的紧凑化。 在多个基准测试中的实验表明，DistilTS可以达到与全尺寸时间序列基础模型相当的预测性能，同时将参数减少了多达1/150，并且推理速度提高了最多6000倍。代码可在[这里](https://this.http URL)获得。
URL
https://arxiv.org/abs/2601.12785
