Abstract
Existing retrieval-augmented approaches for Dense Video Captioning (DVC) often fail to achieve accurate temporal segmentation aligned with true event boundaries, as they rely on heuristic strategies that overlook ground truth event boundaries. The proposed framework, \textbf{STaRC}, overcomes this limitation by supervising frame-level saliency through a highlight detection module. Note that the highlight detection module is trained on binary labels derived directly from DVC ground truth annotations without the need for additional annotation. We also propose to utilize the saliency scores as a unified temporal signal that drives retrieval via saliency-guided segmentation and informs caption generation through explicit Saliency Prompts injected into the decoder. By enforcing saliency-constrained segmentation, our method produces temporally coherent segments that align closely with actual event transitions, leading to more accurate retrieval and contextually grounded caption generation. We conduct comprehensive evaluations on the YouCook2 and ViTT benchmarks, where STaRC achieves state-of-the-art performance across most of the metrics. Our code is available at this https URL
Abstract (translated)
现有的密集视频字幕（Dense Video Captioning，简称 DVC）增强检索方法常常难以实现与真实事件边界准确对齐的时序分割，因为它们依赖于忽视了地面真实事件边界的启发式策略。所提出的框架**STaRC**通过帧级显著性监督模块克服了这一限制，该模块用于检测亮点。需要注意的是，这个亮点检测模块是基于从 DVC 地面真相注释中直接得出的二进制标签进行训练的，并且不需要额外标注。 我们还建议利用显著性得分作为统一的时间信号，通过显著性引导分割来驱动检索，并通过显式注入解码器中的显著性提示（Saliency Prompts）为字幕生成提供信息。通过强制执行显著性约束的分割，我们的方法能够产生与实际事件转换紧密对齐的时间一致性片段，从而实现更准确的检索和基于上下文的字幕生成。 我们在 YouCook2 和 ViTT 基准测试上进行了全面评估，STaRC 在大多数指标上取得了最先进的性能。代码可在 [此处](此链接需您提供具体URL) 获取。
URL
https://arxiv.org/abs/2603.11460
