Abstract
Processing data at high speeds is becoming increasingly critical as digital economies generate enormous data. The current paradigms for timely data processing are edge computing and data stream processing (DSP). Edge computing places resources closer to where data is generated, while stream processing analyzes the unbounded high-speed data in motion. However, edge stream processing faces rapid workload fluctuations, complicating resource provisioning. Inadequate resource allocation leads to bottlenecks, whereas excess allocation results in wastage. Existing reactive methods, such as threshold-based policies and queuing theory scale only after performance degrades, potentially violating SLAs. Although reinforcement learning (RL) offers a proactive approach through agents that learn optimal runtime adaptation policies, it requires extensive simulation. Furthermore, predictive machine learning models face online distribution and concept drift that minimize their accuracy. We propose a three-step solution to the proactive edge stream processing autoscaling problem. Firstly, a GRU neural network forecasts the upstream load using real-world and synthetic DSP datasets. Secondly, a transfer learning framework integrates the predictive model into an online stream processing system using the DTW algorithm and joint distribution adaptation to handle the disparities between offline and online domains. Finally, a horizontal autoscaling module dynamically adjusts the degree of operator parallelism, based on predicted load while considering edge resource constraints. The lightweight GRU model for load predictions recorded up to 1.3\% SMAPE value on a real-world data set. It outperformed CNN, ARIMA, and Prophet on the SMAPE and RMSE evaluation metrics, with lower training time than the computationally intensive RL models.
Abstract (translated)
高速处理数据变得越来越重要，因为数字经济产生了大量数据。目前及时处理这些数据的方法主要包括边缘计算和数据流处理（DSP）。边缘计算将资源部署在靠近数据生成的地方，而数据流处理则分析未定义边界且速度极高的流动数据。然而，边缘流处理面临的工作负载波动迅速，给资源分配带来了复杂性。不适当的资源配置会导致瓶颈，而过度配置则会造成浪费。现有的反应式方法（例如基于阈值的策略和排队理论）仅在性能下降后才会进行扩展，这可能会违反服务水平协议（SLA）。虽然强化学习（RL）通过代理学习最优运行时自适应策略提供了前瞻性的方式，但这种方法需要大量的模拟工作。此外，在线分布和概念漂移会降低预测性机器学习模型的准确性。 我们提出了一种三步解决方案来解决前瞻性的边缘流处理自动扩展问题。首先，使用GRU神经网络基于现实世界及合成DSP数据集对未来负载进行预测。其次，采用转移学习框架将预测模型集成到在线流处理系统中，并利用DTW算法和联合分布适应性技术处理离线与在线领域之间的差异。最后，一个水平的自动扩展模块会根据预测的工作量动态调整操作员并行度，同时考虑到边缘资源限制。 轻量级的GRU模型在现实世界的数据集上实现了高达1.3% SMAPE值（对称平均百分比误差）的表现，并且在SMAPE和RMSE评估指标上优于CNN、ARIMA及Prophet模型。该模型还具有比计算密集型RL模型更短的训练时间。
URL
https://arxiv.org/abs/2507.14597
