Abstract
Recent progress in Multimodal Large Language Models (MLLMs) has unlocked powerful cross-modal reasoning abilities, but also raised new safety concerns, particularly when faced with adversarial multimodal inputs. To improve the safety of MLLMs during inference, we introduce a modular and adaptive inference-time intervention technology, AutoSteer, without requiring any fine-tuning of the underlying model. AutoSteer incorporates three core components: (1) a novel Safety Awareness Score (SAS) that automatically identifies the most safety-relevant distinctions among the model's internal layers; (2) an adaptive safety prober trained to estimate the likelihood of toxic outputs from intermediate representations; and (3) a lightweight Refusal Head that selectively intervenes to modulate generation when safety risks are detected. Experiments on LLaVA-OV and Chameleon across diverse safety-critical benchmarks demonstrate that AutoSteer significantly reduces the Attack Success Rate (ASR) for textual, visual, and cross-modal threats, while maintaining general abilities. These findings position AutoSteer as a practical, interpretable, and effective framework for safer deployment of multimodal AI systems.
Abstract (translated)
近期在多模态大型语言模型（MLLMs）方面取得的进展解锁了强大的跨模态推理能力，但同时也引发了新的安全问题，特别是在面对对抗性多模态输入时。为了提高MLLMs在推断过程中的安全性，我们引入了一种模块化且适应性强的推理干预技术AutoSteer，并且无需对基础模型进行微调。AutoSteer包含三个核心组件：（1）一种新颖的安全感知得分（Safety Awareness Score, SAS），它能够自动识别模型内部层之间最相关的安全区分；（2）一个经过训练以估计中间表示产生有害输出可能性的自适应安全探测器；（3）一个轻量级的拒绝头，当检测到安全风险时，它可以有针对性地介入来调节生成过程。在LLaVA-OV和Chameleon模型上进行的一系列安全性关键基准测试表明，AutoSteer显著降低了文本、视觉以及跨模态威胁的攻击成功率（ASR），同时保持了一般能力不变。这些发现将AutoSteer定位为一种实用、可解释且有效的框架，用于多模态AI系统的安全部署。
URL
https://arxiv.org/abs/2507.13255
