Abstract
In this paper a doubly attentive transformer machine translation model (DATNMT) is presented in which a doubly-attentive transformer decoder normally joins spatial visual features obtained via pretrained convolutional neural networks, conquering any gap between image captioning and translation. In this framework, the transformer decoder figures out how to take care of source-language words and parts of an image freely by methods for two separate attention components in an Enhanced Multi-Head Attention Layer of doubly attentive transformer, as it generates words in the target language. We find that the proposed model can effectively exploit not just the scarce multimodal machine translation data, but also large general-domain text-only machine translation corpora, or image-text image captioning corpora. The experimental results show that the proposed doubly-attentive transformer-decoder performs better than a single-decoder transformer model, and gives the state-of-the-art results in the English-German multimodal machine translation task.
Abstract (translated)
本文提出了一种双重细心的变压器机器翻译模型（DATNMT），其中双重注意变压器解码器通常加入通过预训练卷积神经网络获得的空间视觉特征，克服图像字幕和翻译之间的任何差距。在这个框架中，变换器解码器通过双重注意力变换器的增强型多头注意层中的两个单独的注意组件的方法，自由地计算如何自由地处理源语言单词和图像的一部分，因为它在目标语言。我们发现所提出的模型不仅可以有效地利用稀缺的多模式机器翻译数据，而且可以有效地利用大型通用域文本机器翻译语料库或图像文本图像字幕语料库。实验结果表明，所提出的双重注意变压器 - 解码器的性能优于单解码器变压器模型，并在英德多模式机器翻译任务中给出了最先进的结果。
URL
https://arxiv.org/abs/1807.11605
