Abstract
Open-vocabulary object detectors such as Grounding DINO are trained on vast and diverse data, achieving remarkable performance on challenging datasets. Due to that, it is unclear where to find their limitations, which is of major concern when using in safety-critical applications. Real-world data does not provide sufficient control, required for a rigorous evaluation of model generalization. In contrast, synthetically generated data allows to systematically explore the boundaries of model competence/generalization. In this work, we address two research questions: 1) Can we challenge open-vocabulary object detectors with generated image content? 2) Can we find systematic failure modes of those models? To address these questions, we design two automated pipelines using stable diffusion to inpaint unusual objects with high diversity in semantics, by sampling multiple substantives from WordNet and ChatGPT. On the synthetically generated data, we evaluate and compare multiple open-vocabulary object detectors as well as a classical object detector. The synthetic data is derived from two real-world datasets, namely LostAndFound, a challenging out-of-distribution (OOD) detection benchmark, and the NuImages dataset. Our results indicate that inpainting can challenge open-vocabulary object detectors in terms of overlooking objects. Additionally, we find a strong dependence of open-vocabulary models on object location, rather than on object semantics. This provides a systematic approach to challenge open-vocabulary models and gives valuable insights on how data could be acquired to effectively improve these models.
Abstract (translated)
开放式词汇对象检测器（如Grounding DINO）在海量且多样化的数据上进行训练，因此它们在具有挑战性的数据集上表现出色。然而，这种优秀的性能也使得很难找到这些模型的局限性，这在安全关键的应用中尤为重要。现实世界的数据无法提供严格的评估所需的那种控制，而合成生成的数据则允许系统地探索模型能力与泛化边界的极限。 在这项工作中，我们探讨了两个研究问题： 1) 我们能否用生成的内容挑战开放式词汇对象检测器？ 2) 我们能否找到这些模型的系统性失败模式？ 为了解决这些问题，我们设计了两条自动化流水线，使用Stable Diffusion来对合成图像中的异常物体进行多样化的语义填充。我们在两个真实世界的数据集（LostAndFound和NuImages）的基础上生成合成数据，并评估多个开放式词汇对象检测器以及经典的物体检测器。 我们的实验结果表明，修补技术可以挑战开放式词汇模型在忽略对象方面的能力。此外，我们发现这些开放式词汇模型对对象位置的依赖性远远大于语义上的依赖性。 这提供了一种系统化的方法来挑战开放词汇模型，并为如何获取有效的改进数据提供了宝贵的见解。
URL
https://arxiv.org/abs/2506.23751
