Abstract
In this study, a human evaluation is carried out on how hyperparameter settings impact the quality of Transformer-based Neural Machine Translation (NMT) for the low-resourced English--Irish pair. SentencePiece models using both Byte Pair Encoding (BPE) and unigram approaches were appraised. Variations in model architectures included modifying the number of layers, evaluating the optimal number of heads for attention and testing various regularisation techniques. The greatest performance improvement was recorded for a Transformer-optimized model with a 16k BPE subword model. Compared with a baseline Recurrent Neural Network (RNN) model, a Transformer-optimized model demonstrated a BLEU score improvement of 7.8 points. When benchmarked against Google Translate, our translation engines demonstrated significant improvements. Furthermore, a quantitative fine-grained manual evaluation was conducted which compared the performance of machine translation systems. Using the Multidimensional Quality Metrics (MQM) error taxonomy, a human evaluation of the error types generated by an RNN-based system and a Transformer-based system was explored. Our findings show the best-performing Transformer system significantly reduces both accuracy and fluency errors when compared with an RNN-based model.
Abstract (translated)
在这项研究中，我们对如何调整超参数对低资源英语-爱尔兰对之间的Transformer基神经机器翻译（NMT）的质量进行了人类评估。使用了Byte Pair Encoding（BPE）和unigram方法的两个SentencePiece模型进行评估。模型架构的变异包括修改层数，评估注意力机制的最佳头数以及尝试各种正则化技术。在用16k个BPE子词模型的Transformer优化模型中记录了最大的性能改进。与基循环神经网络（RNN）模型相比，Transformer优化模型显示BLEU得分提高了7.8分。与谷歌翻译进行基准测试时，我们的翻译引擎表现出了显著的改进。此外，进行了一项定量的细粒度手动评估，比较了机器翻译系统的性能。使用多维质量度量（MQM）错误分类器，研究了基于RNN和Transformer的系统生成的错误类型的性能。我们的研究结果表明，与基于RNN的模型相比，性能最佳的Transformer系统在比较时显著减少了准确性和流畅性误差。
URL
https://arxiv.org/abs/2403.02366
