Abstract
We propose a deep learning framework designed to significantly optimize bandwidth for motion-transfer-enabled video applications, including video conferencing, virtual reality interactions, health monitoring systems, and vision-based real-time anomaly detection. To capture complex motion effectively, we utilize the First Order Motion Model (FOMM), which encodes dynamic objects by detecting keypoints and their associated local affine transformations. These keypoints are identified using a self-supervised keypoint detector and arranged into a time series corresponding to the successive frames. Forecasting is performed on these keypoints by integrating two advanced generative time series models into the motion transfer pipeline, namely the Variational Recurrent Neural Network (VRNN) and the Gated Recurrent Unit with Normalizing Flow (GRU-NF). The predicted keypoints are subsequently synthesized into realistic video frames using an optical flow estimator paired with a generator network, thereby facilitating accurate video forecasting and enabling efficient, low-frame-rate video transmission. We validate our results across three datasets for video animation and reconstruction using the following metrics: Mean Absolute Error, Joint Embedding Predictive Architecture Embedding Distance, Structural Similarity Index, and Average Pair-wise Displacement. Our results confirm that by utilizing the superior reconstruction property of the Variational Autoencoder, the VRNN integrated FOMM excels in applications involving multi-step ahead forecasts such as video conferencing. On the other hand, by leveraging the Normalizing Flow architecture for exact likelihood estimation, and enabling efficient latent space sampling, the GRU-NF based FOMM exhibits superior capabilities for producing diverse future samples while maintaining high visual quality for tasks like real-time video-based anomaly detection.
Abstract (translated)
我们提出了一种深度学习框架，旨在显著优化运动传输增强型视频应用的带宽使用，包括视频会议、虚拟现实交互、健康监测系统以及基于视觉的实时异常检测。为了有效捕捉复杂动作，我们采用了第一顺序运动模型（FOMM），该模型通过检测关键点及其相关的局部仿射变换来编码动态对象。 这些关键点通过自监督的关键点检测器识别，并根据连续帧排列成时间序列。接下来，在运动传输管道中结合了两种先进的生成式时间序列模型——变分递归神经网络（VRNN）和门控循环单元与规范化流（GRU-NF），对这些关键点进行预测。 随后，利用光流估算器配合生成网络将预测的关键点合成出逼真的视频帧。这不仅促进了准确的视频预测，还使得低帧率视频传输成为可能，并且更加高效。我们在三个用于视频动画和重建的数据集上通过以下指标验证了我们的结果：平均绝对误差、关节嵌入预测架构嵌入距离、结构相似性指数以及平均成对位移。 实验结果显示，利用变分自编码器的优越重构特性，VRNN集成FOMM在多步预测应用（如视频会议）中表现出色。而通过采用规范化流架构进行精确似然估计，并实现高效的潜在空间采样，基于GRU-NF的FOMM则展现了更强大的能力，在实时基于视频的异常检测等任务中生成多样化的未来样本同时保持高质量视觉效果。
URL
https://arxiv.org/abs/2504.05537
