Abstract
Vision-Language Models (VLMs) have achieved impressive progress in multimodal text generation, yet their rapid adoption raises increasing concerns about security vulnerabilities. Existing backdoor attacks against VLMs primarily rely on explicit pixel-level triggers or imperceptible perturbations injected into images. While effective, these approaches reduce stealthiness and remain vulnerable to image-based defenses. We introduce concept-guided backdoor attacks, a new paradigm that operates at the semantic concept level rather than on raw pixels. We propose two different attacks. The first, Concept-Thresholding Poisoning (CTP), uses explicit concepts in natural images as triggers: only samples containing the target concept are poisoned, causing the model to behave normally in all other cases but consistently inject malicious outputs whenever the concept appears. The second, CBL-Guided Unseen Backdoor (CGUB), leverages a Concept Bottleneck Model (CBM) during training to intervene on internal concept activations, while discarding the CBM branch at inference time to keep the VLM unchanged. This design enables systematic replacement of a targeted label in generated text (for example, replacing "cat" with "dog"), even when the replacement behavior never appears in the training data. Experiments across multiple VLM architectures and datasets show that both CTP and CGUB achieve high attack success rates while maintaining moderate impact on clean-task performance. These findings highlight concept-level vulnerabilities as a critical new attack surface for VLMs.
Abstract (translated)
视觉-语言模型（VLM）在多模态文本生成方面取得了显著进展，但其快速采用引发了越来越多的安全漏洞担忧。现有的针对VLM的后门攻击主要依赖于显式的像素级触发器或不可察觉地注入到图像中的细微扰动。尽管这些方法有效，它们降低了隐蔽性，并且仍然容易受到基于图像的防御措施的影响。我们引入了一种新的概念引导型后门攻击范式，在语义概念层面而非原始像素上操作。 我们提出了两种不同的攻击方式： 1. **概念阈值投毒（Concept-Thresholding Poisoning, CTP）**：使用自然图像中的显式概念作为触发器，仅对包含目标概念的样本进行投毒。在其他情况下模型正常运行，但一旦出现特定概念时则始终注入恶意输出。 2. **基于概念瓶颈引导的不可见后门攻击（CBL-Guided Unseen Backdoor, CGUB）**：利用训练过程中的概念瓶颈模型（CBM），干预内部的概念激活，并在推理阶段丢弃CBM分支以保持VLM不变。这种设计使得即使替代行为从未出现在训练数据中，也能够系统地替换生成文本中的目标标签（例如将“猫”替换成“狗”）。 跨多个VLM架构和数据集的实验表明，CTP 和 CGUB 都能在保持清洁任务性能适中影响的同时实现较高的攻击成功率。这些发现突显了概念层面的漏洞作为VLM面临的新重要攻击面的重要性。
URL
https://arxiv.org/abs/2512.00713
