NMT
-
Quantity vs. Quality of Monolingual Source Data in Automatic Text Translation: Can It Be Too Little If It Is Too Good?
2024-10-17 17:20:40Idris Abdulmumin, Bashir Shehu Galadanci, Garba Aliyu, Shamsuddeen Hassan MuhammadAbstract
Monolingual data, being readily available in large quantities, has been used to upscale the scarcely available parallel data to train better models for automatic translation. Self-learning, where a model is made to learn from its output, is one approach to exploit such data. However, it has been shown that too much of this data can be detrimental to the performance of the model if the available parallel data is comparatively extremely low. In this study, we investigate whether the monolingual data can also be too little and if this reduction, based on quality, has any effect on the performance of the translation model. Experiments have shown that on English-German low-resource NMT, it is often better to select only the most useful additional data, based on quality or closeness to the domain of the test data, than utilizing all of the available data.
Abstract (translated)
URL
https://arxiv.org/abs/2410.13783
PDF
-
Enhancing Assamese NLP Capabilities: Introducing a Centralized Dataset Repository
2024-10-15 05:26:57S. Tamang, D. J. BoraAbstract
This paper introduces a centralized, open-source dataset repository designed to advance NLP and NMT for Assamese, a low-resource language. The repository supports various tasks like sentiment analysis, named entity recognition, and machine translation by providing both pre-training and fine-tuning corpora. We review existing datasets, highlighting the need for standardized resources in Assamese NLP, and discuss potential applications in AI-driven research, such as LLMs, OCR, and chatbots. While promising, challenges like data scarcity and linguistic diversity remain. The repository aims to foster collaboration and innovation, promoting Assamese language research in the digital age.
Abstract (translated)
URL
https://arxiv.org/abs/2410.11291
PDF
-
QE-EBM: Using Quality Estimators as Energy Loss for Machine Translation
2024-10-14 07:39:33Gahyun Yoo, Jay Yoon LeeAbstract
Reinforcement learning has shown great promise in aligning language models with human preferences in a variety of text generation tasks, including machine translation. For translation tasks, rewards can easily be obtained from quality estimation (QE) models which can generate rewards for unlabeled data. Despite its usefulness, reinforcement learning cannot exploit the gradients with respect to the QE score. We propose QE-EBM, a method of employing quality estimators as trainable loss networks that can directly backpropagate to the NMT model. We examine our method on several low and high resource target languages with English as the source language. QE-EBM outperforms strong baselines such as REINFORCE and proximal policy optimization (PPO) as well as supervised fine-tuning for all target languages, especially low-resource target languages. Most notably, for English-to-Mongolian translation, our method achieves improvements of 2.5 BLEU, 7.1 COMET-KIWI, 5.3 COMET, and 6.4 XCOMET relative to the supervised baseline.
Abstract (translated)
URL
https://arxiv.org/abs/2410.10228
PDF
-
ChakmaNMT: A Low-resource Machine Translation On Chakma Language
2024-10-14 07:21:31Aunabil Chakma, Aditya Chakma, Soham Khisa, Chumui Tripura, Masum Hasan, Rifat ShahriyarAbstract
The geopolitical division between the indigenous Chakma population and mainstream Bangladesh creates a significant cultural and linguistic gap, as the Chakma community, mostly residing in the hill tracts of Bangladesh, maintains distinct cultural traditions and language. Developing a Machine Translation (MT) model or Chakma to Bangla could play a crucial role in alleviating this cultural-linguistic divide. Thus, we have worked on MT between CCP-BN(Chakma-Bangla) by introducing a novel dataset of 15,021 parallel samples and 42,783 monolingual samples of the Chakma Language. Moreover, we introduce a small set for Benchmarking containing 600 parallel samples between Chakma, Bangla, and English. We ran traditional and state-of-the-art models in NLP on the training set, where fine-tuning BanglaT5 with back-translation using transliteration of Chakma achieved the highest BLEU score of 17.8 and 4.41 in CCP-BN and BN-CCP respectively on the Benchmark Dataset. As far as we know, this is the first-ever work on MT for the Chakma Language. Hopefully, this research will help to bridge the gap in linguistic resources and contribute to preserving endangered languages. Our dataset link and codes will be published soon.
Abstract (translated)
URL
https://arxiv.org/abs/2410.10219
PDF
-
NusaMT-7B: Machine Translation for Low-Resource Indonesian Languages with Large Language Models
Abstract
Large Language Models (LLMs) have demonstrated exceptional promise in translation tasks for high-resource languages. However, their performance in low-resource languages is limited by the scarcity of both parallel and monolingual corpora, as well as the presence of noise. Consequently, such LLMs suffer with alignment and have lagged behind State-of-The-Art (SoTA) neural machine translation (NMT) models in these settings. This paper introduces NusaMT-7B, an LLM-based machine translation model for low-resource Indonesian languages, starting with Balinese and Minangkabau. Leveraging the pretrained LLaMA2-7B, our approach integrates continued pre-training on monolingual data, Supervised Fine-Tuning (SFT), self-learning, and an LLM-based data cleaner to reduce noise in parallel sentences. In the FLORES-200 multilingual translation benchmark, NusaMT-7B outperforms SoTA models in the spBLEU metric by up to +6.69 spBLEU in translations into Balinese and Minangkabau, but underperforms by up to -3.38 spBLEU in translations into higher-resource languages. Our results show that fine-tuned LLMs can enhance translation quality for low-resource languages, aiding in linguistic preservation and cross-cultural communication.
Abstract (translated)
URL
https://arxiv.org/abs/2410.07830
PDF
-
Optimizing the Training Schedule of Multilingual NMT using Reinforcement Learning
2024-10-08 15:20:13Alexis Allemann, \`Alex R. Atrio, Andrei Popescu-BelisAbstract
Multilingual NMT is a viable solution for translating low-resource languages (LRLs) when data from high-resource languages (HRLs) from the same language family is available. However, the training schedule, i.e. the order of presentation of languages, has an impact on the quality of such systems. Here, in a many-to-one translation setting, we propose to apply two algorithms that use reinforcement learning to optimize the training schedule of NMT: (1) Teacher-Student Curriculum Learning and (2) Deep Q Network. The former uses an exponentially smoothed estimate of the returns of each action based on the loss on monolingual or multilingual development subsets, while the latter estimates rewards using an additional neural network trained from the history of actions selected in different states of the system, together with the rewards received. On a 8-to-1 translation dataset with LRLs and HRLs, our second method improves BLEU and COMET scores with respect to both random selection of monolingual batches and shuffled multilingual batches, by adjusting the number of presentations of LRL vs. HRL batches.
Abstract (translated)
URL
https://arxiv.org/abs/2410.06118
PDF
-
On Instruction-Finetuning Neural Machine Translation Models
2024-10-07 23:26:13Vikas Raunak, Roman Grundkiewicz, Marcin Junczys-DowmuntAbstract
In this work, we introduce instruction finetuning for Neural Machine Translation (NMT) models, which distills instruction following capabilities from Large Language Models (LLMs) into orders-of-magnitude smaller NMT models. Our instruction-finetuning recipe for NMT models enables customization of translations for a limited but disparate set of translation-specific tasks. We show that NMT models are capable of following multiple instructions simultaneously and demonstrate capabilities of zero-shot composition of instructions. We also show that through instruction finetuning, traditionally disparate tasks such as formality-controlled machine translation, multi-domain adaptation as well as multi-modal translations can be tackled jointly by a single instruction finetuned NMT model, at a performance level comparable to LLMs such as GPT-3.5-Turbo. To the best of our knowledge, our work is among the first to demonstrate the instruction-following capabilities of traditional NMT models, which allows for faster, cheaper and more efficient serving of customized translations.
Abstract (translated)
URL
https://arxiv.org/abs/2410.05553
PDF
-
Can the Variation of Model Weights be used as a Criterion for Self-Paced Multilingual NMT?
2024-10-05 12:52:51\`Alex R. Atrio, Alexis Allemann, Ljiljana Dolamic, Andrei Popescu-BelisAbstract
Many-to-one neural machine translation systems improve over one-to-one systems when training data is scarce. In this paper, we design and test a novel algorithm for selecting the language of minibatches when training such systems. The algorithm changes the language of the minibatch when the weights of the model do not evolve significantly, as measured by the smoothed KL divergence between all layers of the Transformer network. This algorithm outperforms the use of alternating monolingual batches, but not the use of shuffled batches, in terms of translation quality (measured with BLEU and COMET) and convergence speed.
Abstract (translated)
许多对一神经机器翻译系统在训练数据有限的情况下表现优于一对一系统。在本文中，我们设计了一种新算法来选择在训练过程中语言，该算法在模型权重在平滑的KL散度定义的Transformer网络的所有层之间时没有显著变化时更改minibatch的语言。与交替单语料库相比，该算法在翻译质量（使用BLEU和COMET）和收敛速度方面都表现出色。而与洗牌单语料库相比，该算法的表现并不突出。
URL
https://arxiv.org/abs/2410.04147
PDF
-
Efficient Technical Term Translation: A Knowledge Distillation Approach for Parenthetical Terminology Translation
2024-10-01 13:40:28Jiyoon Myung, Jihyeon Park, Jungki Son, Kyungro Lee, Joohyung HanAbstract
This paper addresses the challenge of accurately translating technical terms, which are crucial for clear communication in specialized fields. We introduce the Parenthetical Terminology Translation (PTT) task, designed to mitigate potential inaccuracies by displaying the original term in parentheses alongside its translation. To implement this approach, we generated a representative PTT dataset using a collaborative approach with large language models and applied knowledge distillation to fine-tune traditional Neural Machine Translation (NMT) models and small-sized Large Language Models (sLMs). Additionally, we developed a novel evaluation metric to assess both overall translation accuracy and the correct parenthetical presentation of terms. Our findings indicate that sLMs did not consistently outperform NMT models, with fine-tuning proving more effective than few-shot prompting, particularly in models with continued pre-training in the target language. These insights contribute to the advancement of more reliable terminology translation methodologies.
Abstract (translated)
本文解决了在专业领域中准确翻译技术术语的挑战，这些术语对于清晰沟通至关重要。我们引入了Parenthetical Terminology Translation（PTT）任务，旨在通过在原文术语旁边显示其翻译来缓解潜在的不准确性。为了实现这种方法，我们使用合作方法生成了一个PTT数据集，并使用知识蒸馏对传统的神经机器翻译（NMT）模型和小型语言模型（sLMs）进行微调。此外，我们还开发了一种新的评估指标，以评估总翻译准确性以及术语的正确父式呈现。我们的研究结果表明，sLMs并没有始终优于NMT模型，而微调证明比少量的提示更有效，特别是在目标语言中持续预训练的模型中。这些见解有助于推动更可靠术语翻译方法的发展。
URL
https://arxiv.org/abs/2410.00683
PDF
-
Contrastive Token Learning with Similarity Decay for Repetition Suppression in Machine Translation
2024-09-30 02:21:39Huangyu Dai, Ben Chen, Kaidi Chen, Ying Han, Zihan Liang, Wen JiangAbstract
For crosslingual conversation and trade, Neural Machine Translation (NMT) is pivotal yet faces persistent challenges with monotony and repetition in generated content. Traditional solutions that rely on penalizing text redundancy or token reoccurrence have shown limited efficacy, particularly for lengthy article and e-commerce descriptions with inherent redundancy, even with the advent of Large Language Models (LLMs). This paper investigates the underlying causes of textual repetition through the lens of information entropy, attributing the phenomenon to the elevated uncertainty within the input text. To address this, a novel algorithm named Contrastive Token Learning with Similarity Decay (CTSD) is introduced, which modulates the suppression of tokens dynamically, informed by varying attention weights and inter-token distances. Furthermore, an e-commerce dataset comprised of title texts of online real items is compiled and released susceptible to hallucination translations to benchmark the algorithm. Extensive evaluations demonstrate that CTSD significantly outperforms existing approaches in precision and generalizability. Additional online A/B testing underscores its practical value, showing marked improvements in user engagement and conversion. Notably, this method has been implemented with full traffic on eight multilingual sites of this http URL, the largest B2B e-commerce platform in the world.
Abstract (translated)
为了进行跨语言对话和贸易，神经机器翻译（NMT）在生成内容的单调化和重复性方面具有关键作用，但生成的内容中存在持续的挑战。传统解决方案依赖于惩罚文本冗余或词项再出现，对于具有固有冗余的长篇文章和电子商务描述，效果有限，即使在大型语言模型（LLMs）出现的情况下。本文通过信息熵的视角研究了文本重复的潜在原因，将这种现象归因于输入文本中的不确定性增加。为解决这个问题，提出了一种名为Contrastive Token Learning with Similarity Decay（CTSD）的新算法，该算法动态地调节词项的抑制，受到不同注意权重和词项间距离的影响。此外，还编译和发布了一个电子商务数据集，包括在线实物的标题文本，以评估算法的性能。大量评估结果表明，CTSD在精度和泛化能力方面显著优于现有方法。进一步的在线A/B测试证实了其实用价值，表明在用户参与度和转化率方面取得了显著的改善。值得注意的是，这种方法在八个多语言网站的全面流量下实现，这是世界上最大的B2B电子商务平台。
URL
https://arxiv.org/abs/2409.19877
PDF
-
Do We Need Domain-Specific Embedding Models? An Empirical Investigation
Abstract
Embedding models play a crucial role in representing and retrieving information across various NLP applications. Recent advancements in Large Language Models (LLMs) have further enhanced the performance of embedding models, which are trained on massive amounts of text covering almost every domain. These models are often benchmarked on general-purpose datasets like Massive Text Embedding Benchmark (MTEB), where they demonstrate superior performance. However, a critical question arises: Is the development of domain-specific embedding models necessary when general-purpose models are trained on vast corpora that already include specialized domain texts? In this paper, we empirically investigate this question, choosing the finance domain as an example. We introduce the Finance Massive Text Embedding Benchmark (FinMTEB), a counterpart to MTEB that consists of financial domain-specific text datasets. We evaluate the performance of seven state-of-the-art embedding models on FinMTEB and observe a significant performance drop compared to their performance on MTEB. To account for the possibility that this drop is driven by FinMTEB's higher complexity, we propose four measures to quantify dataset complexity and control for this factor in our analysis. Our analysis provides compelling evidence that state-of-the-art embedding models struggle to capture domain-specific linguistic and semantic patterns, even when trained on large general-purpose corpora. This study sheds light on the necessity of developing domain-specific embedding models in the LLM era, offering valuable insights for researchers and practitioners.
Abstract (translated)
嵌入模型在各种自然语言处理应用中扮演着至关重要的角色，尤其是在大型语言模型（LLMs） recent 进步的加强作用下。这些模型在训练过程中使用大量覆盖几乎所有领域的文本数据，通常在通用数据集上进行比较，表现出优异的性能。然而，一个关键的问题出现了：在训练通用模型以涵盖庞大文本数据集的同时，是否需要开发特定领域的嵌入模型？在本文中，我们通过实证研究来回答这个问题，并选择金融领域作为例子。我们引入了金融百万文本嵌入基准（FinMTEB），这是 MTEB 的一个对应物，由金融领域的特定文本数据组成。我们评估了七个最先进的嵌入模型在 FinMTEB 上的性能，观察到它们的性能明显低于它们在 MTEB 上的性能。为了考虑到 FinMTEB 更高的复杂性，我们提出了四个度量来量化数据集复杂度，并控制这个因素在我们的分析中。我们的分析提供了令人信服的证据，即使训练在大规模通用数据集上，最先进的嵌入模型也无法捕捉到领域特定的语言和语义模式。这项研究阐明了在 LLM 时代开发领域特定嵌入模型的必要性，为研究人员和实践者提供了宝贵的洞见。
URL
https://arxiv.org/abs/2409.18511
PDF
-
Artificial Data Point Generation in Clustered Latent Space for Small Medical Datasets
2024-09-26 09:51:08Yasaman Haghbin, Hadi Moradi, Reshad HosseiniAbstract
One of the growing trends in machine learning is the use of data generation techniques, since the performance of machine learning models is dependent on the quantity of the training dataset. However, in many medical applications, collecting large datasets is challenging due to resource constraints, which leads to overfitting and poor generalization. This paper introduces a novel method, Artificial Data Point Generation in Clustered Latent Space (AGCL), designed to enhance classification performance on small medical datasets through synthetic data generation. The AGCL framework involves feature extraction, K-means clustering, cluster evaluation based on a class separation metric, and the generation of synthetic data points from clusters with distinct class representations. This method was applied to Parkinson's disease screening, utilizing facial expression data, and evaluated across multiple machine learning classifiers. Experimental results demonstrate that AGCL significantly improves classification accuracy compared to baseline, GN and kNNMTD. AGCL achieved the highest overall test accuracy of 83.33% and cross-validation accuracy of 90.90% in majority voting over different emotions, confirming its effectiveness in augmenting small datasets.
Abstract (translated)
机器学习的一个不断增长的趋势是使用数据生成技术，因为机器学习模型的性能取决于训练数据集的数量。然而，在许多医疗应用中，收集大量数据是非常具有挑战性的，由于资源限制，导致过拟合和欠拟合。本文介绍了一种名为人工数据点生成在聚类潜在空间（AGCL）的新方法，旨在通过合成数据生成来提高小型医疗数据的分类性能。AGCL框架包括特征提取、K-means聚类、基于类分离指标的聚类评估和从聚类中生成合成数据点。将该方法应用于帕金森病筛查，利用面部表情数据，并在多个机器学习分类器上进行评估。实验结果表明，AGCL相对于基线、GN和kNNMTD显著提高了分类准确率。AGCL在不同的情感上的整体测试准确率为83.33%，交叉验证准确率为90.90%，证实了其在增强小数据集的有效性。
URL
https://arxiv.org/abs/2409.17685
PDF
-
Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimization
2024-09-26 09:32:12Kaden Uhlig, Joern Wuebker, Raphael Reinauer, John DeNeroAbstract
Reinforcement Learning from Human Feedback (RLHF) and derivative techniques like Direct Preference Optimization (DPO) are task-alignment algorithms used to repurpose general, foundational models for specific tasks. We show that applying task-alignment to neural machine translation (NMT) addresses an existing task--data mismatch in NMT, leading to improvements across all languages of a multilingual model, even when task-alignment is only applied to a subset of those languages. We do so by introducing Direct Quality Optimization (DQO), a variant of DPO leveraging a pre-trained translation quality estimation model as a proxy for human preferences, and verify the improvements with both automatic metrics and human evaluation.
Abstract (translated)
强化学习来自人类反馈（RLHF）和像直接偏好优化（DPO）这样的导数技术是用于将通用基础模型适应特定任务的任务对齐算法。我们证明了将任务对齐应用于自然语言翻译（NMT）解决了NMT中存在的任务-数据不匹配问题，从而在多语言模型的所有语言上改善了任务完成情况。为此，我们引入了直接质量优化（DQO），一种利用预训练翻译质量估计模型作为人类偏好的代理，并通过自动指标和人类评估来验证改进。
URL
https://arxiv.org/abs/2409.17673
PDF
-
Multilingual Transfer and Domain Adaptation for Low-Resource Languages of Spain
2024-09-24 09:46:27Yuanchang Luo, Zhanglin Wu, Daimeng Wei, Hengchao Shang, Zongyao Li, Jiaxin Guo, Zhiqiang Rao, Shaojun Li, Jinlong Yang, Yuhao Xie, Jiawei Zheng Bin Wei, Hao YangAbstract
This article introduces the submission status of the Translation into Low-Resource Languages of Spain task at (WMT 2024) by Huawei Translation Service Center (HW-TSC). We participated in three translation tasks: spanish to aragonese (es-arg), spanish to aranese (es-arn), and spanish to asturian (es-ast). For these three translation tasks, we use training strategies such as multilingual transfer, regularized dropout, forward translation and back translation, labse denoising, transduction ensemble learning and other strategies to neural machine translation (NMT) model based on training deep transformer-big architecture. By using these enhancement strategies, our submission achieved a competitive result in the final evaluation.
Abstract (translated)
这篇文章介绍了华为翻译服务中心（HW-TSC）在2024年机器翻译国际会议（WMT 2024）上对将西班牙语翻译成低资源语言任务的提交状态。我们参与了三个翻译任务：西班牙语到阿拉贡语（es-arg）、西班牙语到阿拉尼亚语（es-arn）和西班牙语到 Asturian（es-ast）。为了这三个翻译任务，我们使用了一些训练策略，如多语言迁移、正则化失活、前向翻译和反向翻译、labse denoising、转换器集成学习等策略，对基于训练大架构的神经机器翻译（NMT）模型进行优化。通过使用这些增强策略，我们的提交在最终评估中获得了竞争力的结果。
URL
https://arxiv.org/abs/2409.15924
PDF
-
Exploring the traditional NMT model and Large Language Model for chat translation
2024-09-24 08:48:25Jinlong Yang, Hengchao Shang, Daimeng Wei, Jiaxin Guo, Zongyao Li, Zhanglin Wu, Zhiqiang Rao, Shaojun Li, Yuhao Xie, Yuanchang Luo, Jiawei Zheng, Bin Wei, Hao YangAbstract
This paper describes the submissions of Huawei Translation Services Center(HW-TSC) to WMT24 chat translation shared task on English$\leftrightarrow$Germany (en-de) bidirection. The experiments involved fine-tuning models using chat data and exploring various strategies, including Minimum Bayesian Risk (MBR) decoding and self-training. The results show significant performance improvements in certain directions, with the MBR self-training method achieving the best results. The Large Language Model also discusses the challenges and potential avenues for further research in the field of chat translation.
Abstract (translated)
本文描述了华为翻译服务中心（HW-TSC）在英语到德国（en-de）双向交互对话数据集上参加WMT24聊天翻译共享任务的研究提交。实验过程中，使用了聊天数据对模型进行微调并探讨了各种策略，包括最小贝叶斯风险（MBR）解码和自训练。结果表明，在某些方向上取得了显著的性能提升，其中MBR自训练方法取得了最佳结果。大型语言模型还讨论了聊天翻译领域进一步研究的挑战和潜在方向。
URL
https://arxiv.org/abs/2409.16331
PDF
-
HW-TSC's Submission to the CCMT 2024 Machine Translation Tasks
2024-09-23 09:20:19Zhanglin Wu, Yuanchang Luo, Daimeng Wei, Jiawei Zheng, Bin Wei, Zongyao Li, Hengchao Shang, Jiaxin Guo, Shaojun Li, Weidong Zhang, Ning Xie, Hao YangAbstract
This paper presents the submission of Huawei Translation Services Center (HW-TSC) to machine translation tasks of the 20th China Conference on Machine Translation (CCMT 2024). We participate in the bilingual machine translation task and multi-domain machine translation task. For these two translation tasks, we use training strategies such as regularized dropout, bidirectional training, data diversification, forward translation, back translation, alternated training, curriculum learning, and transductive ensemble learning to train neural machine translation (NMT) models based on the deep Transformer-big architecture. Furthermore, to explore whether large language model (LLM) can help improve the translation quality of NMT systems, we use supervised fine-tuning to train llama2-13b as an Automatic post-editing (APE) model to improve the translation results of the NMT model on the multi-domain machine translation task. By using these plyometric strategies, our submission achieves a competitive result in the final evaluation.
Abstract (translated)
本文提交了华为翻译服务中心（HW-TSC）在2024年第三届中国机器翻译会议（CCMT 2024）上对机器翻译任务的提交。我们参与了双语机器翻译任务和多领域机器翻译任务。为了训练这两种翻译任务，我们采用了诸如正则化dropout、双向训练、数据多样性、前向翻译、反向翻译、交替训练、课程学习以及转换器集成学习等训练策略，基于深度Transformer-big架构训练神经机器翻译（NMT）模型。 此外，为了探讨大型语言模型（LLM）是否可以帮助提高神经机器翻译（NMT）系统的翻译质量，我们使用有监督的微调来训练llama2-13b作为自动后编辑（APE）模型，以提高NMT模型在多领域机器翻译任务上的翻译结果。通过使用这些跳涨策略，我们的提交在最终评估中实现了竞争力的结果。
URL
https://arxiv.org/abs/2409.14842
PDF
-
Choose the Final Translation from NMT and LLM hypotheses Using MBR Decoding: HW-TSC's Submission to the WMT24 General MT Shared Task
2024-09-23 08:25:37Zhanglin Wu, Daimeng Wei, Zongyao Li, Hengchao Shang, Jiaxin Guo, Shaojun Li, Zhiqiang Rao, Yuanchang Luo, Ning Xie, Hao YangAbstract
This paper presents the submission of Huawei Translate Services Center (HW-TSC) to the WMT24 general machine translation (MT) shared task, where we participate in the English to Chinese (en2zh) language pair. Similar to previous years' work, we use training strategies such as regularized dropout, bidirectional training, data diversification, forward translation, back translation, alternated training, curriculum learning, and transductive ensemble learning to train the neural machine translation (NMT) model based on the deep Transformer-big architecture. The difference is that we also use continue pre-training, supervised fine-tuning, and contrastive preference optimization to train the large language model (LLM) based MT model. By using Minimum Bayesian risk (MBR) decoding to select the final translation from multiple hypotheses for NMT and LLM-based MT models, our submission receives competitive results in the final evaluation.
Abstract (translated)
本文介绍了华为翻译服务中心（HW-TSC）参与MT24通用机器翻译（MT）共享任务，我们参加英语到中文（en2zh）语言对。与以往的工作类似，我们使用如正则化 dropout、双向训练、数据多样化、前向翻译、反向翻译、交替训练、课程学习以及转换器模型（NMT）等训练策略来训练基于深度Transformer架构的神经机器翻译（NMT）模型。不同之处在于，我们还使用了继续预训练、有监督的微调以及对比偏好优化来训练基于MT的大语言模型（LLM）模型。通过使用最小贝叶斯风险（MBR）解码来选择NMT和LLM-based MT模型的最终翻译，我们在最终评估中获得了竞争力的结果。
URL
https://arxiv.org/abs/2409.14800
PDF
-
EMMeTT: Efficient Multimodal Machine Translation Training
2024-09-20 14:03:23Piotr \.Zelasko, Zhehuai Chen, Mengru Wang, Daniel Galvez, Oleksii Hrinchuk, Shuoyang Ding, Ke Hu, Jagadeesh Balam, Vitaly Lavrukhin, Boris GinsburgAbstract
A rising interest in the modality extension of foundation language models warrants discussion on the most effective, and efficient, multimodal training approach. This work focuses on neural machine translation (NMT) and proposes a joint multimodal training regime of Speech-LLM to include automatic speech translation (AST). We investigate two different foundation model architectures, decoder-only GPT and encoder-decoder T5, extended with Canary-1B's speech encoder. To handle joint multimodal training, we propose a novel training framework called EMMeTT. EMMeTT improves training efficiency with the following: balanced sampling across languages, datasets, and modalities; efficient sequential data iteration; and a novel 2D bucketing scheme for multimodal data, complemented by a batch size optimizer (OOMptimizer). We show that a multimodal training consistently helps with both architectures. Moreover, SALM-T5 trained with EMMeTT retains the original NMT capability while outperforming AST baselines on four-language subsets of FLORES and FLEURS. The resultant Multimodal Translation Model produces strong text and speech translation results at the same time.
Abstract (translated)
对基础语言模型扩展模式的日益关注需要讨论最有效、最有效的多模态训练方法。本文重点关注神经机器翻译（NMT），并提出了一个结合Speech-LLM的联合多模态训练方案，包括自动语音翻译（AST）。我们研究了两种不同的基础模型架构：仅解码器GPT和编码器-解码器T5，用Canary-1B的语音编码器进行了扩展。为了处理联合多模态训练，我们提出了一个名为EMMeTT的新型训练框架。EMMeTT通过以下方式提高了训练效率：平衡跨语言、数据集和模态的采样；高效的序列数据迭代；以及一种新颖的2D桶位方案，由批量大小优化器（OOMptimizer）补充。我们证明了多模态训练在两种架构中都是有效的。此外，使用EMMeTT训练的SALM-T5保留了原始NMT能力，同时在FLORES和FLEURS的四个语言子集上优于AST基线。所得到的跨模态翻译模型可以在同一时间产生强文本和语音翻译结果。
URL
https://arxiv.org/abs/2409.13523
PDF
-
Thesis proposal: Are We Losing Textual Diversity to Natural Language Processing?
Abstract
This thesis argues that the currently widely used Natural Language Processing algorithms possibly have various limitations related to the properties of the texts they handle and produce. With the wide adoption of these tools in rapid progress, we must ask what these limitations are and what are the possible implications of integrating such tools even more deeply into our daily lives. As a testbed, we have chosen the task of Neural Machine Translation (NMT). Nevertheless, we aim for general insights and outcomes, applicable even to current Large Language Models (LLMs). We ask whether the algorithms used in NMT have inherent inductive biases that are beneficial for most types of inputs but might harm the processing of untypical texts. To explore this hypothesis, we define a set of measures to quantify text diversity based on its statistical properties, like uniformity or rhythmicity of word-level surprisal, on multiple scales (sentence, discourse, language). We then conduct a series of experiments to investigate whether NMT systems struggle with maintaining the diversity of such texts, potentially reducing the richness of the language generated by these systems, compared to human translators. We search for potential causes of these limitations rooted in training objectives and decoding algorithms. Our ultimate goal is to develop alternatives that do not enforce uniformity in the distribution of statistical properties in the output and that allow for better global planning of the translation, taking into account the intrinsic ambiguity of the translation task.
Abstract (translated)
本论文认为，目前广泛使用的自然语言处理算法可能存在与它们处理和产生的文本相关的各种局限性。随着这些工具的广泛采用，我们必须问一问这些局限性是什么，以及将这样的工具更深入地融入我们的日常生活中可能产生的后果。作为实验台，我们选择了自然机器翻译（NMT）作为任务。然而，我们的目标是获取一般性的见解和结论，这些结论对目前的较大语言模型（LLMs）也适用。我们问NMT中使用的算法是否具有固有的归纳偏见，对大多数类型的输入有益，但对不规则文本的加工造成损害。为了探讨这个假设，我们定义了一系列指标，用于量化基于统计特征文本的多样性，如单词级别的出人意料的统一性或节奏性，在多个尺度（句子、段落、语言）上进行。然后我们进行了一系列实验，以调查NMT系统是否难以维持这种文本多样性，从而减少由这些系统生成的语言的丰富程度，与人类翻译者相比。我们寻找导致这些局限性的潜在原因，源于训练目标和解码算法的目标。我们最终的目标是开发出不需要在输出和分布的统计特性上实现统一性的替代方案，允许更好的全局规划翻译，考虑到翻译任务的固有歧义性。
URL
https://arxiv.org/abs/2409.09568
PDF
-
Vision-fused Attack: Advancing Aggressive and Stealthy Adversarial Text against Neural Machine Translation
2024-09-08 08:22:17Yanni Xue, Haojie Hao, Jiakai Wang, Qiang Sheng, Renshuai Tao, Yu Liang, Pu Feng, Xianglong LiuAbstract
While neural machine translation (NMT) models achieve success in our daily lives, they show vulnerability to adversarial attacks. Despite being harmful, these attacks also offer benefits for interpreting and enhancing NMT models, thus drawing increased research attention. However, existing studies on adversarial attacks are insufficient in both attacking ability and human imperceptibility due to their sole focus on the scope of language. This paper proposes a novel vision-fused attack (VFA) framework to acquire powerful adversarial text, i.e., more aggressive and stealthy. Regarding the attacking ability, we design the vision-merged solution space enhancement strategy to enlarge the limited semantic solution space, which enables us to search for adversarial candidates with higher attacking ability. For human imperceptibility, we propose the perception-retained adversarial text selection strategy to align the human text-reading mechanism. Thus, the finally selected adversarial text could be more deceptive. Extensive experiments on various models, including large language models (LLMs) like LLaMA and GPT-3.5, strongly support that VFA outperforms the comparisons by large margins (up to 81%/14% improvements on ASR/SSIM).
Abstract (translated)
虽然神经机器翻译（NMT）模型在我们的日常生活中取得了成功，但它们对对抗性攻击非常脆弱。尽管这些攻击是有害的，但它们也为解释和增强 NMT 模型提供了好处，因此吸引了更多的研究关注。然而，现有的对抗性攻击研究由于仅关注语言范围而不足，因此本文提出了一种新颖的视频融合攻击（VFA）框架，以获取强大的对抗性文本，即更具有攻击性和隐秘性的文本。在攻击能力方面，我们设计了愿景融合解决方案空间增强策略，以扩大有限语义解决方案空间，这使我们能够搜索具有更高攻击能力的对抗性候选者。在人类感知方面，我们提出了保留感知的人类文本选择策略，使人类的文本阅读机制与攻击策略保持一致。因此，最终选择的攻击性文本可能更具有欺骗性。在各种模型（包括大型语言模型，如LLMs（如LLLM和GPT-3.5））的广泛实验中，VFA 的优越性明显超过基于ASR（自动文本评分）和SSIM（结构从文本到序列匹配）的比较（ASR/SSIM改进率高达81%/14%）。
URL
https://arxiv.org/abs/2409.05021
PDF
