Abstract
Dense Video Captioning (DVC) is a challenging multimodal task that involves temporally localizing multiple events within a video and describing them with natural language. While query-based frameworks enable the simultaneous, end-to-end processing of localization and captioning, their reliance on shared queries often leads to significant multi-task interference between the two tasks, as well as temporal redundancy in localization. In this paper, we propose utilizing role-specific queries that separate localization and captioning into independent components, allowing each to exclusively learn its role. We then employ contrastive alignment to enforce semantic consistency between the corresponding outputs, ensuring coherent behavior across the separated queries. Furthermore, we design a novel suppression mechanism in which mutual temporal overlaps across queries are penalized to tackle temporal redundancy, supervising the model to learn distinct, non-overlapping event regions for more precise localization. Additionally, we introduce a lightweight module that captures core event concepts to further enhance semantic richness in captions through concept-level representations. We demonstrate the effectiveness of our method through extensive experiments on major DVC benchmarks YouCook2 and ActivityNet Captions.
Abstract (translated)
密集视频字幕（Dense Video Captioning，DVC）是一项具有挑战性的多模态任务，涉及在视频中定位多个事件并用自然语言描述它们。虽然基于查询的框架能够同时、端到端地处理定位和字幕生成这两个任务，但这些方法通常依赖于共享查询，这往往会导致两个任务之间存在显著的任务干扰以及定位过程中出现的时间冗余。在这篇论文中，我们提出使用角色特定查询的方法，将定位与字幕生成分离开来作为独立的组件，使每个组件能够专注于学习其自身的功能。随后，我们采用对比对齐方法以强制执行对应输出之间的语义一致性，确保分离后的查询之间的一致行为。此外，我们设计了一种新颖的抑制机制，在此机制中跨查询的时间重叠被惩罚，用以解决时间冗余问题，并监督模型为更精准定位学习到不相交的独特事件区域。另外，我们还引入了一个轻量级模块来捕捉核心事件概念，通过概念级别的表示进一步增强字幕中的语义丰富性。 我们在主要的DVC基准测试YouCook2和ActivityNet Captions上进行了广泛的实验，并展示了我们的方法的有效性。
URL
https://arxiv.org/abs/2603.11439
