Abstract
Recent Vision-Language Models (VLMs) have made remarkable progress in multimodal understanding tasks, yet their evaluation on long video understanding remains unreliable. Due to limited frame inputs, key frames necessary for answering the question may be missing from the model's input. However, models that truthfully refuse to answer under such uncertainty are marked as incorrect, while those that guess may coincidentally produce the correct answer and thus obtain deceptively higher accuracy, leading to misleading evaluation results and encouraging models to guess rather than respond honestly. To address this issue, we introduce VirtueBench, a benchmark explicitly designed to assess model trustworthiness under uncertainty. VirtueBench constructs multiple frame-sampling levels for each video and provides ground truths that distinguish between answerable and unanswerable cases. Evaluations on 25 open-source and commercial VLMs reveal distinct refusal behaviors across different model families, with refusal accuracy ranging from over 70% in the best models to nearly 0% in the worst. Moreover, most models exhibit a substantial drop in refusal when the prompt does not explicitly require them to do so. These findings highlight the need for developing trustworthy VLMs for multimodal understanding, guided by benchmarks and leaderboards that emphasize reliability and trustworthiness.
Abstract (translated)
最近的视觉-语言模型（VLM）在多模态理解任务中取得了显著进展，但在处理长视频理解时其评估仍不可靠。由于输入帧数量有限，关键帧可能未能被纳入模型的输入之中，而这些关键帧正是回答问题所需的。然而，在这种不确定性下，那些诚实地拒绝作答的模型会被认为是错误的，相反地，那些猜测答案的模型可能会偶然给出正确答案，并因此获得不实的高准确率评价，导致误导性的评估结果并鼓励模型去猜而不是诚实回应。为了解决这一问题，我们引入了VirtueBench，这是一个专门用于在不确定性环境下评估模型可信度的基准测试。 VirtueBench 为每个视频构建多个帧采样层级，并提供明确区分可回答和不可回答情况的事实依据。对25种开源及商业视觉-语言模型进行评测的结果显示，不同模型家族的表现差异显著，拒绝准确率从最佳模型超过70%到最差模型接近于零不等。此外，在提示未明示要求它们必须拒绝作答的情况下，大多数模型的拒绝率都会大幅下降。 这些发现强调了开发可靠的视觉-语言模型对于多模态理解的重要性，并指出了需要通过注重可靠性和可信度的基准测试和排行榜来引导这一发展方向。
URL
https://arxiv.org/abs/2603.07071
