Abstract
We address the problem of video question answering (video QA) with temporal grounding in a weakly supervised setup, without any temporal annotations. Given a video and a question, we generate an open-ended answer grounded with the start and end time. For this task, we propose TOGA: a vision-language model for Temporally Grounded Open-Ended Video QA with Weak Supervision. We instruct-tune TOGA to jointly generate the answer and the temporal grounding. We operate in a weakly supervised setup where the temporal grounding annotations are not available. We generate pseudo labels for temporal grounding and ensure the validity of these labels by imposing a consistency constraint between the question of a grounding response and the response generated by a question referring to the same temporal segment. We notice that jointly generating the answers with the grounding improves performance on question answering as well as grounding. We evaluate TOGA on grounded QA and open-ended QA tasks. For grounded QA, we consider the NExT-GQA benchmark which is designed to evaluate weakly supervised grounded question answering. For open-ended QA, we consider the MSVD-QA and ActivityNet-QA benchmarks. We achieve state-of-the-art performance for both tasks on these benchmarks.
Abstract (translated)
我们解决的是在弱监督环境下，视频问答（video QA）中的时间定位问题，而无需任何时间标注。给定一个视频和一个问题时，我们会生成基于开始时间和结束时间的开放式答案。为了解决这个任务，我们提出了TOGA：一种用于弱监督下的、具有时间定位的开放性视频问答的视觉-语言模型(Temporally Grounded Open-Ended Video QA)。通过指令微调，TOGA能够同时生成答案和时间定位信息。我们在时间标注不可用的弱监督环境中操作，并为时间定位生成伪标签，以确保这些标签的有效性，我们通过对同一时间段的问题回答之间的一致性约束来实现这一点。 我们注意到，与单独生成答案或时间定位相比，同时生成二者可以提高问答及定位性能。我们在有依据的问答和开放式问答任务上评估TOGA的表现。对于基于证据的问答，我们考虑NExT-GQA基准测试，该测试旨在评估弱监督下的基于证据的问题回答能力。而对于开放性问答，我们参考MSVD-QA和ActivityNet-QA这两个基准。在这些基准测试中，我们的方法在两项任务上均取得了最先进的性能表现。
URL
https://arxiv.org/abs/2506.09445
