Abstract
Parameter-Efficient transfer learning with Adapters have been studied in Natural Language Processing (NLP) as an alternative to full fine-tuning. Adapters are memory-efficient and scale well with downstream tasks by training small bottle-neck layers added between transformer layers while keeping the large pretrained language model (PLMs) frozen. In spite of showing promising results in NLP, these methods are under-explored in Information Retrieval. While previous studies have only experimented with dense retriever or in a cross lingual retrieval scenario, in this paper we aim to complete the picture on the use of adapters in IR. First, we study adapters for SPLADE, a sparse retriever, for which adapters not only retain the efficiency and effectiveness otherwise achieved by finetuning, but are memory-efficient and orders of magnitude lighter to train. We observe that Adapters-SPLADE not only optimizes just 2\% of training parameters, but outperforms fully fine-tuned counterpart and existing parameter-efficient dense IR models on IR benchmark datasets. Secondly, we address domain adaptation of neural retrieval thanks to adapters on cross-domain BEIR datasets and TripClick. Finally, we also consider knowledge sharing between rerankers and first stage rankers. Overall, our study complete the examination of adapters for neural IR
Abstract (translated)
在自然语言处理(NLP)中，使用Adapters作为参数高效的转移学习替代方法已经得到了研究。Adapters能够在Transformer层之间添加小型瓶颈层，同时保持大型预训练语言模型(PLM)冻结，从而实现 Memory-Efficient Transfer Learning。尽管在NLP中取得了 promising 的结果，但在信息检索中这些方法仍然未被深入研究。尽管以前的研究仅尝试过密集检索或跨语言检索场景，但本 paper 旨在完整描述在IR中使用Adapters的情况。首先，我们研究了Adapters-SPLADE，它是一个稀疏检索器，Adapters不仅保留了经过微调后实现的效率与效果，而且具有 Memory-Efficient 和数量级更轻的训练能力。我们观察到Adapters-SPLADE不仅优化了训练参数的2\% ，而且在IR基准数据集上比完全微调的替代品和现有的参数高效的密集IR模型表现更好。其次，我们考虑了跨域BEIR数据和 TripClick Adapters 的神经网络检索域适应问题。最后，我们还考虑了重新排名器和第一级排名器之间的知识共享。总之，我们的研究涵盖了神经网络IRAdapters的使用。
URL
https://arxiv.org/abs/2303.13220
