Abstract
Recent advances in vision-language models (VLMs) trained on web-scale image-text pairs have enabled impressive zero-shot transfer across a diverse range of visual tasks. However, comprehensive and independent evaluation beyond standard benchmarks is essential to understand their robustness, limitations, and real-world applicability. This paper presents a systematic evaluation framework for VLMs under natural adversarial scenarios for diverse downstream tasks, which has been overlooked in previous evaluation works. We evaluate a wide range of VLMs (CLIP, robust CLIP, BLIP2, and SigLIP2) on curated adversarial datasets (typographic attacks, ImageNet-A, and natural language-induced adversarial examples). We measure the natural adversarial performance of selected VLMs for zero-shot image classification, semantic segmentation, and visual question answering. Our analysis reveals that robust CLIP models can amplify natural adversarial vulnerabilities, and CLIP models significantly reduce performance for natural language-induced adversarial examples. Additionally, we provide interpretable analyses to identify failure modes. We hope our findings inspire future research in robust and fair multimodal pattern recognition.
Abstract (translated)
基于大规模网络图文对训练的视觉语言模型（VLMs）在多种视觉任务上展现出令人瞩目的零样本迁移能力。然而，超越标准基准的全面独立评估对于理解其鲁棒性、局限性和实际应用价值至关重要。本研究提出了一种针对自然对抗场景下多样化下游任务的VLM系统评估框架，该框架在既往评估工作中被忽视。我们在一系列精心构建的对抗数据集（包括文字攻击、ImageNet-A及自然语言诱导的对抗样本）上，对CLIP、鲁棒CLIP、BLIP2和SigLIP2等主流VLMs进行了广泛评估。通过测量所选VLMs在零样本图像分类、语义分割及视觉问答任务中的自然对抗性能，分析发现：鲁棒CLIP模型可能放大自然对抗脆弱性，而CLIP模型对自然语言诱导的对抗样本性能显著下降。此外，我们提供了可解释性分析以识别失效模式。希望本研究能启发未来关于鲁棒公平的多模态模式识别研究。
URL
https://arxiv.org/abs/2604.04473
