Abstract
We propose a realistic scenario for the unsupervised video learning where neither task boundaries nor labels are provided when learning a succession of tasks. We also provide a non-parametric learning solution for the under-explored problem of unsupervised video continual learning. Videos represent a complex and rich spatio-temporal media information, widely used in many applications, but which have not been sufficiently explored in unsupervised continual learning. Prior studies have only focused on supervised continual learning, relying on the knowledge of labels and task boundaries, while having labeled data is costly and not practical. To address this gap, we study the unsupervised video continual learning (uVCL). uVCL raises more challenges due to the additional computational and memory requirements of processing videos when compared to images. We introduce a general benchmark experimental protocol for uVCL by considering the learning of unstructured video data categories during each task. We propose to use the Kernel Density Estimation (KDE) of deep embedded video features extracted by unsupervised video transformer networks as a non-parametric probabilistic representation of the data. We introduce a novelty detection criterion for the incoming new task data, dynamically enabling the expansion of memory clusters, aiming to capture new knowledge when learning a succession of tasks. We leverage the use of transfer learning from the previous tasks as an initial state for the knowledge transfer to the current learning task. We found that the proposed methodology substantially enhances the performance of the model when successively learning many tasks. We perform in-depth evaluations on three standard video action recognition datasets, including UCF101, HMDB51, and Something-to-Something V2, without using any labels or class boundaries.
Abstract (translated)
我们提出了一种无监督视频学习的真实场景，在这种场景下，当连续执行多项任务时，既没有提供任务边界也没有提供标签。此外，针对未充分研究的无监督视频持续学习问题，我们也提供了一个非参数化的解决方案。视频代表了复杂且丰富的时空媒体信息，在许多应用中被广泛使用，但在无监督持续学习领域尚未得到充分探索。先前的研究仅关注有监督的持续学习，并依赖于标签和任务边界的知识，而获得标注数据的成本高昂且不切实际。为了解决这一缺口，我们研究了无监督视频连续学习（uVCL）。相较于处理图像，由于视频处理在计算和内存需求方面的要求更高，因此uVCL提出了更大的挑战。 为了便于对uVCL的研究，我们引入了一种通用的基准实验协议，该协议考虑到了每个任务中未结构化的视频数据类别的学习。我们建议使用无监督视频变换网络提取的深度嵌入式视频特征来进行核密度估计（KDE），以作为非参数化概率数据表示方法。我们还提出了一个新颖性检测标准，用于判断新任务数据，并动态启用内存簇扩展，旨在捕捉在连续任务学习过程中的新知识。 我们利用从先前任务中转移的学习成果作为当前学习任务的知识传递的初始状态。研究发现，所提出的这种方法显著提升了模型在连续执行多项任务时的表现。我们在三个标准视频动作识别数据集上进行了深入评估，包括UCF101、HMDB51和Something-to-Something V2，且未使用任何标签或类别边界信息。
URL
https://arxiv.org/abs/2508.21773
