Abstract
Recently, video-language understanding has achieved great success through large-scale pre-training. However, data scarcity remains a prevailing challenge. This study quantitatively reveals an "impossible trinity" among data quantity, diversity, and quality in pre-training datasets. Recent efforts seek to refine large-scale, diverse ASR datasets compromised by low quality through synthetic annotations. These methods successfully leverage useful information in multimodal video content (frames, tags, ASR transcripts, etc.) to refine the original annotations. Nevertheless, they struggle to mitigate noise within synthetic annotations and lack scalability as the dataset size expands. To address these issues, we introduce the Video DataFlywheel framework, which iteratively refines video annotations with improved noise control methods. For iterative refinement, we first leverage a video-language model to generate synthetic annotations, resulting in a refined dataset. Then, we pre-train on it and fine-tune on human refinement examples for a stronger model. These processes are repeated for continuous improvement. For noise control, we present AdaTaiLr, a novel noise control method that requires weaker assumptions on noise distribution, thereby proving more effective in large datasets with theoretical guarantees. The combination of iterative refinement and AdaTaiLr can achieve better scalability in video-language understanding. Extensive experiments show that our framework outperforms existing data refinement baselines, delivering a 3% performance boost and improving dataset quality with minimal diversity loss. Furthermore, our refined dataset facilitates significant improvements in various video-language understanding tasks, including video question answering and text-video retrieval.
Abstract (translated)
近年来，通过大规模预训练，视频语言理解取得了巨大的成功。然而，数据稀缺仍然是一个持续的挑战。本研究在预训练数据集上揭示了数据量、多样性和质量之间的“不可能三角”关系。为了通过合成注释来修复质量低的大规模、多样 ASR 数据集，近期的努力尝试利用多模态视频内容的有用信息来优化原始注释。这些方法成功地利用了多模态视频内容（帧、标签、ASR 转录等）中的有用信息来优化原始注释。然而，它们在合成注释内很难消除噪声，并且在数据集规模扩大时缺乏可扩展性。为了应对这些问题，我们引入了 Video DataFlywheel 框架，该框架通过逐步优化带有更好噪声控制方法的视频注释进行迭代改进。对于迭代改进，我们首先利用视频语言模型生成合成注释，从而得到一个优化后的数据集。然后，我们对它进行预训练，并在人类改进实例上进行微调，以获得更强的模型。这些过程会重复进行以持续改进。为了噪声控制，我们提出了 AdaTaiLr，一种新颖的噪声控制方法，它对噪声分布的弱假设，从而在大型数据集上证明更有效的效果。结合迭代改进和 AdaTaiLr，可以在视频语言理解中实现更好的可扩展性。大量实验证明，我们的框架超越了现有的数据修复基线，实现了3%的性能提升，并且在最小多样性损失的情况下提高了数据质量。此外，经过优化的数据集对各种视频语言理解任务都取得了显著的改进，包括视频问答和文本视频检索。
URL
https://arxiv.org/abs/2409.19532
