Abstract
The task of accurate and efficient language translation is an extremely important information processing task. Machine learning enabled and automated translation that is accurate and fast is often a large topic of interest in the machine learning and data science communities. In this study, we examine using local Generative Pretrained Transformer (GPT) models to perform automated zero shot black-box, sentence wise, multi-natural-language translation into English text. We benchmark 16 different open-source GPT models, with no custom fine-tuning, from the Huggingface LLM repository for translating 50 different non-English languages into English using translated TED Talk transcripts as the reference dataset. These GPT model inference calls are performed strictly locally, on single A100 Nvidia GPUs. Benchmark metrics that are reported are language translation accuracy, using BLEU, GLEU, METEOR, and chrF text overlap measures, and wall-clock time for each sentence translation. The best overall performing GPT model for translating into English text for the BLEU metric is ReMM-v2-L2-13B with a mean score across all tested languages of $0.152$, for the GLEU metric is ReMM-v2-L2-13B with a mean score across all tested languages of $0.256$, for the chrF metric is Llama2-chat-AYT-13B with a mean score across all tested languages of $0.448$, and for the METEOR metric is ReMM-v2-L2-13B with a mean score across all tested languages of $0.438$.
Abstract (translated)
准确且高效的机器翻译是一个极其重要的信息处理任务。通过机器学习实现和自动翻译，通常在机器学习和数据科学社区是一个大的研究主题。在这项研究中，我们使用局部生成预训练Transformer（GPT）模型在英语文本上进行自动零样本黑色文本翻译。我们使用翻译的TED演讲转录作为参考数据集，将50种非英语语言翻译成英语。这些GPT模型推理都在本地进行，使用单个A100 Nvidia GPU。报告的基准指标包括语言翻译准确性、BLEU、GLEU、METEOR和chrF文本重叠度衡量，以及每个句子翻译的墙钟时间。在BLEU指标上，翻译成英语文本的最佳GPT模型是ReMM-v2-L2-13B，平均分数为所有测试语言的$0.152$；在GLEU指标上，翻译成英语文本的最佳GPT模型是ReMM-v2-L2-13B，平均分数为所有测试语言的$0.256$；在chrF指标上，Llama2-chat-AYT-13B的平均分数为所有测试语言的$0.448$；在METEOR指标上，ReMM-v2-L2-13B的平均分数为所有测试语言的$0.438$。
URL
https://arxiv.org/abs/2404.14680
