Abstract
Neural retrieval models have acquired significant effectiveness gains over the last few years compared to term-based methods. Nevertheless, those models may be brittle when faced to typos, distribution shifts or vulnerable to malicious attacks. For instance, several recent papers demonstrated that such variations severely impacted models performances, and then tried to train more resilient models. Usual approaches include synonyms replacements or typos injections -- as data-augmentation -- and the use of more robust tokenizers (characterBERT, BPE-dropout). To further complement the literature, we investigate in this paper adversarial training as another possible solution to this robustness issue. Our comparison includes the two main families of BERT-based neural retrievers, i.e. dense and sparse, with and without distillation techniques. We then demonstrate that one of the most simple adversarial training techniques -- the Fast Gradient Sign Method (FGSM) -- can improve first stage rankers robustness and effectiveness. In particular, FGSM increases models performances on both in-domain and out-of-domain distributions, and also on queries with typos, for multiple neural retrievers.
Abstract (translated)
神经网络检索模型在过去几年中相对于术语方法取得了显著的有效性增益。然而,当面临打字错误、分布迁移或受到恶意攻击等问题时,这些模型可能变得脆性。例如,几篇最近的论文表明,这些变化严重影响了模型性能,然后尝试训练更坚韧的模型。通常的方法包括同义词替换或打字错误注入(作为数据增强)以及使用更可靠的分词器(字符BERT、BPE-dropout)。为了进一步补充文献,我们在本文中探讨对抗训练作为解决这个问题的另一个可能解决方案。我们的比较包括基于BERT的神经网络检索的两个主要家族,即密集和稀疏,以及不使用蒸馏技术的两个版本。然后,我们证明其中一个最简单的对抗训练技术——快速梯度签名方法(FGSM)——可以提高第一阶段排名器的稳健性和有效性。特别是,FGSM可以提高多个神经网络检索器的内部和外部分布以及包含打字错误的查询的性能。
URL
https://arxiv.org/abs/2301.10576
