Abstract
Transformer-based architectures have dominated various areas of machine learning in recent years. In this paper, we introduce a novel robust attention mechanism designed to enhance the resilience of transformer-based architectures. Crucially, this technique can be integrated into existing transformers as a plug-and-play layer, improving their robustness without the need for additional training or fine-tuning. Through comprehensive experiments and ablation studies, we demonstrate that our ProTransformer significantly enhances the robustness of transformer models across a variety of prediction tasks, attack mechanisms, backbone architectures, and data domains. Notably, without further fine-tuning, the ProTransformer consistently improves the performance of vanilla transformers by 19.5%, 28.3%, 16.1%, and 11.4% for BERT, ALBERT, DistilBERT, and RoBERTa, respectively, under the classical TextFooler attack. Furthermore, ProTransformer shows promising resilience in large language models (LLMs) against prompting-based attacks, improving the performance of T5 and LLaMA by 24.8% and 17.8%, respectively, and enhancing Vicuna by an average of 10.4% against the Jailbreaking attack. Beyond the language domain, ProTransformer also demonstrates outstanding robustness in both vision and graph domains.
Abstract (translated)
基于 Transformer 的架构近年来在机器学习的各个领域占据了主导地位。本文介绍了一种新颖的鲁棒注意力机制，旨在增强基于 Transformer 架构的韧性。至关重要的是，这项技术可以作为即插即用层集成到现有的 Transformer 中，无需额外训练或微调即可提高其鲁棒性。通过全面的实验和消融研究，我们证明了我们的 ProTransformer 显著增强了在多种预测任务、攻击机制、基础架构和数据域中的 Transformer 模型的鲁棒性。值得注意的是，在经典的 TextFooler 攻击下，无需进一步微调，ProTransformer 一致地提高了普通 Transformer 的性能：BERT 提升 19.5%，ALBERT 提升 28.3%，DistilBERT 提升 16.1%，RoBERTa 提升 11.4%。此外，在大语言模型（LLMs）对抗提示攻击时，ProTransformer 表现出了良好的韧性，提高了 T5 和 LLaMA 的性能分别为 24.8% 和 17.8%，并且在应对 Jailbreaking 攻击时，平均提升了 Vicuna 的性能达 10.4%。除了语言领域之外，在视觉和图领域中，ProTransformer 还展示了出色的鲁棒性。
URL
https://arxiv.org/abs/2410.23182
