Abstract
We introduce a new test set for visual question answering (VQA) called BinaryVQA to push the limits of VQA models. Our dataset includes 7,800 questions across 1,024 images and covers a wide variety of objects, topics, and concepts. For easy model evaluation, we only consider binary questions. Questions and answers are formulated and verified carefully and manually. Around 63% of the questions have positive answers. The median number of questions per image and question length are 7 and 5, respectively. The state of the art OFA model achieves 75% accuracy on BinaryVQA dataset, which is significantly lower than its performance on the VQA v2 test-dev dataset (94.7%). We also analyze the model behavior along several dimensions including: a) performance over different categories such as text, counting and gaze direction, b) model interpretability, c) the effect of question length on accuracy, d) bias of models towards positive answers and introduction of a new score called the ShuffleAcc, and e) sensitivity to spelling and grammar errors. Our investigation demonstrates the difficulty of our dataset and shows that it can challenge VQA models for next few years. Data and code are publicly available at: DATA and CODE.
Abstract (translated)
我们引入了一个新的视觉问答测试集(VQA)称为BinaryVQA,以挑战VQA模型的极限。我们的数据集包括7,800个问题,跨越1,024图像,涵盖了广泛的对象、话题和概念。为了易于模型评估,我们仅考虑二进制问题。问题和答案经过仔细且手动地形式化验证。大约有63%的问题有正面答案。每个图像的提问数和问题长度的中位数分别为7和5。最先进的OFDA模型在BinaryVQA数据集上实现了75%的准确率,这比它在VQA v2测试-dev数据集上的表现(94.7%)差得多。我们还分析模型在不同维度上的行为,包括:a)在不同类别上的表现,如文本、计数和注视方向,b)模型解释能力,c)问题长度对准确性的影响,d)模型倾向于正面回答的偏见,并引入了称为Shuffleacc的新得分,以及e)对拼写和语法错误敏感性。我们的研究展示了我们数据集的难度,并表明它可以在未来几年内挑战VQA模型。数据和代码在数据代码库中公开可用。
URL
https://arxiv.org/abs/2301.12032
