Abstract
Document-level neural machine translation (DNMT) has shown promising results by incorporating more context information. However, this approach also introduces a length bias problem, whereby DNMT suffers from significant translation quality degradation when decoding documents that are much shorter or longer than the maximum sequence length during training. %i.e., the length bias problem. To solve the length bias problem, we propose to improve the DNMT model in training method, attention mechanism, and decoding strategy. Firstly, we propose to sample the training data dynamically to ensure a more uniform distribution across different sequence lengths. Then, we introduce a length-normalized attention mechanism to aid the model in focusing on target information, mitigating the issue of attention divergence when processing longer sequences. Lastly, we propose a sliding window strategy during decoding that integrates as much context information as possible without exceeding the maximum sequence length. The experimental results indicate that our method can bring significant improvements on several open datasets, and further analysis shows that our method can significantly alleviate the length bias problem.
Abstract (translated)
文档级别神经机器翻译（DNMT）通过引入更多的上下文信息表现出良好的效果。然而，这种方法还引入了一个长度偏差问题，即在训练过程中，DNMT会显著降低对长度过短或过长的文档的翻译质量。换句话说，长度偏差问题。为解决长度偏差问题，我们提出了改进DNMT训练方法、注意机制和解码策略。首先，我们动态采样训练数据以保证不同序列长度的数据分布更加均匀。然后，我们引入了一个长度归一化的注意机制，帮助模型集中注意力于目标信息，减轻了在处理长序列时的关注度偏差问题。最后，我们在解码过程中采用滑动窗口策略，在确保最大序列长度的前提下，整合尽可能多的上下文信息。实验结果表明，我们的方法可以在多个公开数据集上带来显著的改进，而进一步的分析显示，我们的方法可以显著减轻长度偏差问题。
URL
https://arxiv.org/abs/2311.11601
