Abstract
Weakly-Supervised Dense Video Captioning aims to localize and describe events in videos trained only on caption annotations, without temporal boundaries. Prior work introduced an implicit supervision paradigm based on Gaussian masking and complementary captioning. However, existing method focuses merely on generating non-overlapping masks without considering their semantic relationship to corresponding events, resulting in simplistic, uniformly distributed masks that fail to capture semantically meaningful regions. Moreover, relying solely on ground-truth captions leads to sub-optimal performance due to the inherent sparsity of existing datasets. In this work, we propose SAIL, which constructs semantically-aware masks through cross-modal alignment. Our similarity aware training objective guides masks to emphasize video regions with high similarity to their corresponding event captions. Furthermore, to guide more accurate mask generation under sparse annotation settings, we introduce an LLM-based augmentation strategy that generates synthetic captions to provide additional alignment signals. These synthetic captions are incorporated through an inter-mask mechanism, providing auxiliary guidance for precise temporal localization without degrading the main objective. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance on both captioning and localization metrics.
Abstract (translated)
弱监督密集视频字幕任务的目标是在仅使用字幕注释（而没有时间边界信息）训练的情况下，定位和描述视频中的事件。先前的工作引入了一种基于高斯掩码和互补性字幕的隐式监督范例。然而，现有的方法仅仅关注生成不重叠的掩码，并未考虑这些掩码与相应事件之间的语义关系，导致生成了简单且均匀分布的掩码，无法捕捉到具有实际意义的语义区域。此外，单纯依赖于真实标签字幕会导致在现有数据集固有的稀疏性下表现不佳。 为了解决这些问题，我们提出了SAIL（Semantic-Aware Implicit Learning）方法，该方法通过跨模态对齐来构建语义感知掩码。我们的相似度感知训练目标引导掩码突出显示视频中与相应事件字幕具有高相似性的区域。此外，为了在标注稀疏的情况下更准确地生成掩码，我们引入了一种基于大规模语言模型（LLM）的增强策略，该策略生成合成字幕以提供额外的对齐信号。这些合成字幕通过一种跨掩码机制被整合进来，在不损害主要目标的前提下提供了精确时间定位的辅助指导。 在ActivityNet Captions和YouCook2数据集上的实验表明，SAIL方法在字幕描述和事件定位两个关键指标上均达到了当前最佳性能。
URL
https://arxiv.org/abs/2603.05437
