Abstract
Text-to-video (T2V) synthesis has advanced rapidly, yet current evaluation metrics primarily capture visual quality and temporal consistency, offering limited insight into how synthetic videos perform in downstream tasks such as text-to-video retrieval (TVR). In this work, we introduce SynTVA, a new dataset and benchmark designed to evaluate the utility of synthetic videos for building retrieval models. Based on 800 diverse user queries derived from MSRVTT training split, we generate synthetic videos using state-of-the-art T2V models and annotate each video-text pair along four key semantic alignment dimensions: Object \& Scene, Action, Attribute, and Prompt Fidelity. Our evaluation framework correlates general video quality assessment (VQA) metrics with these alignment scores, and examines their predictive power for downstream TVR performance. To explore pathways of scaling up, we further develop an Auto-Evaluator to estimate alignment quality from existing metrics. Beyond benchmarking, our results show that SynTVA is a valuable asset for dataset augmentation, enabling the selection of high-utility synthetic samples that measurably improve TVR outcomes. Project page and dataset can be found at this https URL.
Abstract (translated)
文本到视频（T2V）合成技术取得了迅速进展，然而当前的评估指标主要关注于视觉质量和时间一致性，对于下游任务如文本到视频检索（TVR）中合成视频的表现提供了有限的信息。在此项工作中，我们介绍了SynTVA，这是一个全新的数据集和基准测试工具，旨在评估合成视频在构建检索模型方面的实用性。基于从MSRVTT训练集中衍生出的800个多样化用户查询，我们使用最先进的T2V模型生成了合成视频，并对每个视频-文本配对进行了四项关键语义对齐维度上的标注：对象与场景、动作、属性以及提示忠实度。我们的评估框架将通用视频质量评估（VQA）指标与这些对齐分数相关联，并考察它们对于下游TVR性能的预测能力。为了探索扩展路径，我们进一步开发了一个自动评价器来从现有指标中估算对齐质量。除了基准测试之外，我们的结果表明SynTVA是一个有价值的资产，可用于数据集增强，能够选择那些能显著提升TVR成果的高实用性的合成样本。项目页面和数据集可以在此URL找到：[此链接位置应填写具体网址]。
URL
https://arxiv.org/abs/2507.02316
