Abstract
Like many other machine learning applications, neural machine translation (NMT) benefits from over-parameterized deep neural models. However, these models have been observed to be brittle: NMT model predictions are sensitive to small input changes and can show significant variation across re-training or incremental model updates. This work studies a frequently used method in NMT, pseudo-label training (PLT), which is common to the related techniques of forward-translation (or self-training) and sequence-level knowledge distillation. While the effect of PLT on quality is well-documented, we highlight a lesser-known effect: PLT can enhance a model's stability to model updates and input perturbations, a set of properties we call model inertia. We study inertia effects under different training settings and we identify distribution simplification as a mechanism behind the observed results.
Abstract (translated)
与其他机器学习应用一样，神经网络机器翻译(NMT)从过度参数化的深度学习模型中受益。然而，这些模型已被观察到是脆的：NMT模型预测对微小的输入变化非常敏感，并在重新训练或增量模型更新中出现显著差异。这项工作研究了NMT中经常使用的方法——伪标签训练(PLT)，它是 forward-翻译(或自训练)和序列级知识蒸馏相关技术的常见方法。虽然PLT对质量的影响已经被广泛记录，但我们重点介绍了一个不太为人所知的影响：PLT可以增强模型的稳定性，以适应模型更新和输入干扰，我们称之为模型惯性。我们研究了不同训练设置下的惯性效应，并识别分布简化作为观察到结果的机制。
URL
https://arxiv.org/abs/2305.11808
