Abstract
We address the challenge of representation learning from a continuous stream of video as input, in a self-supervised manner. This differs from the standard approaches to video learning where videos are chopped and shuffled during training in order to create a non-redundant batch that satisfies the independently and identically distributed (IID) sample assumption expected by conventional training paradigms. When videos are only available as a continuous stream of input, the IID assumption is evidently broken, leading to poor performance. We demonstrate the drop in performance when moving from shuffled to sequential learning on three tasks: the one-video representation learning method DoRA, standard VideoMAE on multi-video datasets, and the task of future video prediction. To address this drop, we propose a geometric modification to standard optimizers, to decorrelate batches by utilising orthogonal gradients during training. The proposed modification can be applied to any optimizer -- we demonstrate it with Stochastic Gradient Descent (SGD) and AdamW. Our proposed orthogonal optimizer allows models trained from streaming videos to alleviate the drop in representation learning performance, as evaluated on downstream tasks. On three scenarios (DoRA, VideoMAE, future prediction), we show our orthogonal optimizer outperforms the strong AdamW in all three scenarios.
Abstract (translated)
我们解决的是从连续视频流中以自监督方式学习表示的挑战。这与标准的视频学习方法不同，后者在训练过程中将视频片段打乱并重新排序，以便创建一个不冗余且满足独立同分布（IID）样本假设的小批量数据集。当仅能获得连续输入视频时，IID假设显然被破坏，导致性能下降。我们通过三个任务展示了从打乱学习到顺序学习的性能下降：单个视频表示学习方法DoRA、标准VideoMAE在多视频数据集上的应用以及未来视频预测任务。 为了应对这一问题，我们提出了对标准优化器进行几何修改的方法，利用正交梯度在训练过程中去相关化批次。所提出的修改可以应用于任何优化器——我们在随机梯度下降（SGD）和AdamW上进行了演示。我们的正交优化器使从流式视频中训练的模型能够缓解表示学习性能下降的问题，并通过下游任务进行评估。在三个场景（DoRA、VideoMAE、未来预测）下，我们展示了我们的正交优化器在所有三种情况下均优于强大的AdamW。
URL
https://arxiv.org/abs/2504.01961
