Abstract
The rapid growth of video-text data presents challenges in storage and computation during training. Online learning, which processes streaming data in real-time, offers a promising solution to these issues while also allowing swift adaptations in scenarios demanding real-time responsiveness. One strategy to enhance the efficiency and effectiveness of learning involves identifying and prioritizing data that enhances performance on target downstream tasks. We propose Relevance and Specificity-based online filtering framework (ReSpec) that selects data based on four criteria: (i) modality alignment for clean data, (ii) task relevance for target focused data, (iii) specificity for informative and detailed data, and (iv) efficiency for low-latency processing. Relevance is determined by the probabilistic alignment of incoming data with downstream tasks, while specificity employs the distance to a root embedding representing the least specific data as an efficient proxy for informativeness. By establishing reference points from target task data, ReSpec filters incoming data in real-time, eliminating the need for extensive storage and compute. Evaluating on large-scale datasets WebVid2M and VideoCC3M, ReSpec attains state-of-the-art performance on five zeroshot video retrieval tasks, using as little as 5% of the data while incurring minimal compute. The source code is available at this https URL.
Abstract (translated)
视频文本数据的快速增长在训练过程中带来了存储和计算上的挑战。在线学习通过实时处理流式数据，为这些问题提供了一个有前景的解决方案，并且能够在需要实时响应的场景中实现迅速适应。为了提高学习效率和效果，一种策略是识别并优先处理能够提升特定下游任务性能的数据。我们提出了一种基于相关性和具体性的在线过滤框架（ReSpec），该框架根据以下四个标准选择数据：(i) 清晰数据的模态对齐；(ii) 目标聚焦数据的任务关联性；(iii) 对于具有信息量和细节性的数据的具体性；以及(iv) 低延迟处理的效率。相关性通过入流数据与下游任务的概率对齐来确定，而具体性则利用距离代表最低特定度数据的根嵌入作为信息量的有效代理。通过从目标任务数据中建立参考点，ReSpec 实时过滤进入的数据，从而减少了对于大量存储和计算资源的需求。 在大规模数据集WebVid2M 和 VideoCC3M 上进行评估时，即使只使用5% 的数据并且几乎不增加计算成本的情况下，ReSpec 在五个零样本视频检索任务中达到了最先进的性能。源代码可在提供的链接获取。
URL
https://arxiv.org/abs/2504.14875
