Abstract
Spoken dialogue is a primary source of information in videos; therefore, accurately identifying who spoke what and when is essential for deep video understanding. We introduce D-ORCA, a \textbf{d}ialogue-centric \textbf{o}mni-modal large language model optimized for \textbf{r}obust audio-visual \textbf{ca}ptioning. We further curate DVD, a large-scale, high-quality bilingual dataset comprising nearly 40,000 multi-party dialogue videos for training and 2000 videos for evaluation in English and Mandarin, addressing a critical gap in the open-source ecosystem. To ensure fine-grained captioning accuracy, we adopt group relative policy optimization with three novel reward functions that assess speaker attribution accuracy, global speech content accuracy, and sentence-level temporal boundary alignment. These rewards are derived from evaluation metrics widely used in speech processing and, to our knowledge, are applied for the first time as reinforcement learning objectives for audio-visual captioning. Extensive experiments demonstrate that D-ORCA substantially outperforms existing open-source models in speaker identification, speech recognition, and temporal grounding. Notably, despite having only 8 billion parameters, D-ORCA achieves performance competitive with Qwen3-Omni across several general-purpose audio-visual understanding benchmarks. Demos are available at \href{this https URL}{this https URL}. Our code, data, and checkpoints will be available at \href{this https URL}{this https URL}.
Abstract (translated)
口头对话是视频中的主要信息来源，因此准确地识别谁说了什么和什么时候说的对于深入理解视频内容至关重要。我们介绍了D-ORCA，这是一种以对话为中心、面向多模态的大规模语言模型，专门用于鲁棒性的音频-视觉字幕生成（**d**ialogue-centric **o**mni-modal large language model optimized for **r**obust audio-visual **c**aptioning）。此外，我们还策划了DVD数据集，这是一个大规模、高质量的双语数据集，包含近40,000个多方对话视频用于训练以及2000个视频用于评估，这些视频覆盖了英语和普通话。这一数据集填补了开源生态系统中的一个重要空白。 为了确保细粒度字幕的准确性，我们采用了一种基于群体相对策略优化的方法，并引入了三个新颖的奖励函数来评价说话者属性准确率、全球语义内容准确率以及句子级别的时间边界对齐精度。这些奖励函数源自广泛使用的语音处理评估指标，在我们的知识范围内，它们首次被用作音频-视觉字幕生成中的强化学习目标。 通过广泛的实验验证，我们发现D-ORCA在说话人识别、语音识别和时序定位方面大幅优于现有的开源模型。值得注意的是，尽管只有80亿个参数，D-ORCA在多项通用的音频-视频理解基准测试中表现出与Qwen3-Omni相当的性能。 演示可以访问此链接：[this https URL]（请将括号中的内容替换为实际URL）。 我们的代码、数据和检查点将在以下网址提供：[this https URL]（同样，请将括号中的内容替换为实际URL）。
URL
https://arxiv.org/abs/2602.07960
