Abstract
Nearest Neighbor Machine Translation ($k$NN-MT) has achieved great success on domain adaptation tasks by integrating pre-trained Neural Machine Translation (NMT) models with domain-specific token-level retrieval. However, the reasons underlying its success have not been thoroughly investigated. In this paper, we provide a comprehensive analysis of $k$NN-MT through theoretical and empirical studies. Initially, we offer a theoretical interpretation of the working mechanism of $k$NN-MT as an efficient technique to implicitly execute gradient descent on the output projection layer of NMT, indicating that it is a specific case of model fine-tuning. Subsequently, we conduct multi-domain experiments and word-level analysis to examine the differences in performance between $k$NN-MT and entire-model fine-tuning. Our findings suggest that: (1) Incorporating $k$NN-MT with adapters yields comparable translation performance to fine-tuning on in-domain test sets, while achieving better performance on out-of-domain test sets; (2) Fine-tuning significantly outperforms $k$NN-MT on the recall of low-frequency domain-specific words, but this gap could be bridged by optimizing the context representations with additional adapter layers.
Abstract (translated)
Nearest Neighbor Machine Translation ($k$NN-MT) 在跨域任务中取得了巨大的成功，通过将预训练的神经网络机器翻译(NMT)模型与特定域的 token-level 检索相结合，实现了模型微调。然而，其成功的原因并没有得到充分的研究。在本文中，我们通过理论和实证研究提供了 $k$NN-MT 的全面分析。一开始，我们提出了一种理论解释 $k$NN-MT 的工作原理，将其作为在 NMT 输出投影层上隐含执行梯度下降的高效技巧，表明它是模型微调的特定案例。随后，我们进行了跨域实验和单词级别的分析，以检查 $k$NN-MT 和整个模型微调在性能上的差异。我们的发现表明：(1) 将 $k$NN-MT 与适配器相结合可以实现与跨域测试集上的微调相比，在域内测试集上取得相似的翻译性能，但在非域内测试集上取得更好的性能；(2) 微调在低频率域特定单词Recall方面显著优于 $k$NN-MT，但可以通过添加适配器层优化上下文表示来弥平这种差异。
URL
https://arxiv.org/abs/2305.13034
