Abstract
Multimodal large language models have become an important infrastructure for unified processing of visual and linguistic tasks. However, such models are highly susceptible to backdoor implantation during supervised fine-tuning and will steadily output the attacker's predefined harmful responses once a specific trigger pattern is activated. The core challenge of backdoor defense lies in suppressing attack success under low poisoning ratios while preserving the model's normal generation ability. These two objectives are inherently conflicting. Strong suppression often degrades benign performance, whereas weak regularization fails to mitigate backdoor behaviors. To this end, we propose a unified defense framework based on patch augmentation and cross-view regularity, which simultaneously constrains the model's anomalous behaviors in response to triggered patterns from both the feature representation and output distribution levels. Specifically, patch-level data augmentation is combined with cross-view output difference regularization to exploit the fact that backdoor responses are abnormally invariant to non-semantic perturbations and to proactively pull apart the output distributions of the original and perturbed views, thereby significantly suppressing the success rate of backdoor triggering. At the same time, we avoid over-suppression of the model during defense by imposing output entropy constraints, ensuring the quality of normal command generation. Experimental results across three models, two tasks, and six attacks show that our proposed defense method effectively reduces the attack success rate while maintaining a high level of normal text generation capability. Our work enables the secure, controlled deployment of large-scale multimodal models in realistic low-frequency poisoning and covert triggering scenarios.
Abstract (translated)
多模态大语言模型已成为统一处理视觉与语言任务的重要基础设施。然而，此类模型在监督微调过程中极易遭受后门植入，一旦特定触发模式被激活，便会持续输出攻击者预设的有害响应。后门防御的核心挑战在于：在低投毒比例下抑制攻击成功率的同时，需保持模型的正常生成能力——这两个目标本质上是冲突的。强抑制往往损害良性性能，而弱正则化则无法有效缓解后门行为。为此，我们提出一种基于补丁增强与跨视图正则化的统一防御框架，从特征表征与输出分布两个层面同步约束模型对触发模式的异常响应。具体而言，通过结合补丁级数据增强与跨视图输出差异正则化，利用后门响应对非语义扰动呈现异常不变的特性，主动拉大原始视图与扰动视图的输出分布差异，从而显著降低后门触发的成功率。同时，通过施加输出熵约束避免防御过程中对模型的过度抑制，确保正常指令的生成质量。在三个模型、两项任务与六种攻击上的实验结果表明，所提防御方法能在有效降低攻击成功率的同时，维持较高的正常文本生成能力。本研究使大规模多模态模型在现实低频率投毒与隐蔽触发场景下的安全可控部署成为可能。
URL
https://arxiv.org/abs/2604.04488
