NMT
-
Improving the quality of Web-mined Parallel Corpora of Low-Resource Languages using Debiasing Heuristics
2025-02-26 11:56:43Aloka Fernando, Surangika Ranathunga, Nisansa de SilvaAbstract
Parallel Data Curation (PDC) techniques aim to filter out noisy parallel sentences from the web-mined corpora. Prior research has demonstrated that ranking sentence pairs using similarity scores on sentence embeddings derived from Pre-trained Multilingual Language Models (multiPLMs) and training the NMT systems with the top-ranked samples, produces superior NMT performance than when trained using the full dataset. However, previous research has shown that the choice of multiPLM significantly impacts the ranking quality. This paper investigates the reasons behind this disparity across multiPLMs. Using the web-mined corpora CCMatrix and CCAligned for En$\rightarrow$Si, En$\rightarrow$Ta and Si$\rightarrow$Ta, we show that different multiPLMs (LASER3, XLM-R, and LaBSE) are biased towards certain types of sentences, which allows noisy sentences to creep into the top-ranked samples. We show that by employing a series of heuristics, this noise can be removed to a certain extent. This results in improving the results of NMT systems trained with web-mined corpora and reduces the disparity across multiPLMs.
Abstract (translated)
平行数据整理（PDC）技术旨在从网络挖掘的语料库中过滤掉噪声并行句子。先前的研究表明，使用预训练多语言模型（multiPLMs）生成的句子嵌入相似分数对句子对进行排序，并用排名靠前的样本训练神经机器翻译（NMT）系统，比使用完整数据集训练时能获得更优的NMT性能。然而，之前的研究还显示，multiPLM的选择显著影响了排序质量。本文探讨了多语言模型之间存在这种差异的原因。通过使用面向英语到斯洛文尼亚语（En→Si）、英语到泰米尔语（En→Ta）和斯洛文尼亚语到泰米尔语（Si→Ta）的网络挖掘语料库CCMatrix和CCAligned，我们发现不同的multiPLM（如LASER3、XLM-R和LaBSE），在偏好某些类型的句子方面存在偏差，这使得噪声句子能够进入排名靠前的样本中。我们证明通过采用一系列启发式方法可以到一定程度上移除这种噪声，从而改善使用网络挖掘语料库训练的NMT系统的性能，并减少不同multiPLM之间的差异性。
URL
https://arxiv.org/abs/2502.19074
PDF
-
MultiSlav: Using Cross-Lingual Knowledge Transfer to Combat the Curse of Multilinguality
2025-02-20 12:35:25Artur Kot, Miko{\l}aj Koszowski, Wojciech Chojnowski, Mieszko Rutkowski, Artur Nowakowski, Kamil Guttmann, Miko{\l}aj PokrywkaAbstract
Does multilingual Neural Machine Translation (NMT) lead to The Curse of the Multlinguality or provides the Cross-lingual Knowledge Transfer within a language family? In this study, we explore multiple approaches for extending the available data-regime in NMT and we prove cross-lingual benefits even in 0-shot translation regime for low-resource languages. With this paper, we provide state-of-the-art open-source NMT models for translating between selected Slavic languages. We released our models on the HuggingFace Hub (this https URL) under the CC BY 4.0 license. Slavic language family comprises morphologically rich Central and Eastern European languages. Although counting hundreds of millions of native speakers, Slavic Neural Machine Translation is under-studied in our opinion. Recently, most NMT research focuses either on: high-resource languages like English, Spanish, and German - in WMT23 General Translation Task 7 out of 8 task directions are from or to English; massively multilingual models covering multiple language groups; or evaluation techniques.
Abstract (translated)
多语言神经机器翻译（NMT）会导致“多语种的诅咒”还是能够实现语言家族内的跨语言知识转移？在本研究中，我们探讨了多种方法来扩展NMT的数据范围，并证明即使是低资源语言，在零样本翻译模式下也能获得跨语言的好处。通过本文，我们提供了最先进的开源NMT模型，用于选定的斯拉夫语之间的互译。我们在HuggingFace Hub（此链接）以CC BY 4.0许可证发布了我们的模型。斯拉夫语族包括中欧和东欧形态丰富的语言。尽管有数亿母语使用者，但据我们所知，斯拉夫神经机器翻译研究仍处于不足状态。最近的大多数NMT研究要么集中在资源丰富如英语、西班牙语和德语的语言上——在WMT23通用翻译任务中的8个方向中有7个是与英语相关的；覆盖多种语言群体的大规模多语言模型；或者评估技术。
URL
https://arxiv.org/abs/2502.14509
PDF
-
FinMTEB: Finance Massive Text Embedding Benchmark
2025-02-16 04:23:52Yixuan Tang, Yi YangAbstract
Embedding models play a crucial role in representing and retrieving information across various NLP applications. Recent advances in large language models (LLMs) have further enhanced the performance of embedding models. While these models are often benchmarked on general-purpose datasets, real-world applications demand domain-specific evaluation. In this work, we introduce the Finance Massive Text Embedding Benchmark (FinMTEB), a specialized counterpart to MTEB designed for the financial domain. FinMTEB comprises 64 financial domain-specific embedding datasets across 7 tasks that cover diverse textual types in both Chinese and English, such as financial news articles, corporate annual reports, ESG reports, regulatory filings, and earnings call transcripts. We also develop a finance-adapted model, FinPersona-E5, using a persona-based data synthetic method to cover diverse financial embedding tasks for training. Through extensive evaluation of 15 embedding models, including FinPersona-E5, we show three key findings: (1) performance on general-purpose benchmarks shows limited correlation with financial domain tasks; (2) domain-adapted models consistently outperform their general-purpose counterparts; and (3) surprisingly, a simple Bag-of-Words (BoW) approach outperforms sophisticated dense embeddings in financial Semantic Textual Similarity (STS) tasks, underscoring current limitations in dense embedding techniques. Our work establishes a robust evaluation framework for financial NLP applications and provides crucial insights for developing domain-specific embedding models.
Abstract (translated)
嵌入模型在自然语言处理（NLP）的各种应用中用于表示和检索信息，起着至关重要的作用。大型语言模型（LLMs）的最新进展进一步提高了嵌入模型的表现力。尽管这些模型通常是在通用数据集上进行基准测试的，但实际应用却需要领域特定的评估方法。在这项工作中，我们引入了金融大规模文本嵌入基准 (FinMTEB)，它是为金融领域设计的一种专门的 MTEB 对应版本。FinMTEB 包含 64 个专用于金融领域的嵌入数据集，涵盖了从中文和英文中提取的各种文本类型的任务，包括金融新闻文章、公司年报、ESG 报告、监管文件以及收益电话会议记录等。此外，我们开发了一种基于人格的数据合成方法来构建适用于财务的模型 FinPersona-E5，并使用该方法覆盖多样化的金融嵌入任务进行训练。通过对 15 种嵌入模型（包括 FinPersona-E5）进行全面评估，我们展示了三个关键发现：(1) 在通用基准测试中的表现与金融领域的任务关联性有限；(2) 领域适应型模型始终优于其通用对应版本；以及 (3) 出人意料的是，在金融语义文本相似度（STS）任务中，简单的词袋（BoW）方法的表现超过了复杂的密集嵌入技术。这突显了目前在密集嵌入技术中的局限性。我们的工作为金融 NLP 应用建立了稳健的评估框架，并为开发领域特定的嵌入模型提供了关键见解。
URL
https://arxiv.org/abs/2502.10990
PDF
-
Quality-Aware Decoding: Unifying Quality Estimation and Decoding
2025-02-12 16:49:52Sai Koneru, Matthias Huck, Miriam Exel, Jan NiehuesAbstract
An emerging research direction in NMT involves the use of Quality Estimation (QE) models, which have demonstrated high correlations with human judgment and can enhance translations through Quality-Aware Decoding. Although several approaches have been proposed based on sampling multiple candidate translations, none have integrated these models directly into the decoding process. In this paper, we address this by proposing a novel token-level QE model capable of reliably scoring partial translations. We build a uni-directional QE model for this, as decoder models are inherently trained and efficient on partial sequences. We then present a decoding strategy that integrates the QE model for Quality-Aware decoding and demonstrate that the translation quality improves when compared to the N-best list re-ranking with state-of-the-art QE models (upto $1.39$ XCOMET-XXL $\uparrow$). Finally, we show that our approach provides significant benefits in document translation tasks, where the quality of N-best lists is typically suboptimal.
Abstract (translated)
在神经机器翻译（NMT）领域，一个新兴的研究方向是使用质量估计（QE）模型。这些模型已经展示了与人类判断的高度相关性，并且可以通过质量感知解码来提升翻译效果。尽管已有多种方法提出了基于采样多个候选翻译的方案，但还没有任何一种方法将这些模型直接集成到解码过程中。在本文中，我们通过提出一个新颖的、能够在可靠地评分部分翻译上表现优异的分词级别QE模型解决了这一问题。为此，我们构建了一个单向QE模型，因为译码器模型本质上适用于并能高效处理不完整的序列。接着，我们介绍了一种解码策略，该策略将QE模型集成到质量感知解码中，并展示了与使用最先进的QE模型对N-best列表重新排序相比，我们的方法可以提升翻译质量（最多提高1.39个XCOMET-XXL分数）。最后，我们证明了在文档翻译任务中，这种方法提供了显著的优势，在这种情况下，N-best列表的质量通常不理想。
URL
https://arxiv.org/abs/2502.08561
PDF
-
Unsupervised Translation of Emergent Communication
2025-02-11 13:41:06Ido Levy, Orr Paradise, Boaz Carmeli, Ron Meir, Shafi Goldwasser, Yonatan BelinkovAbstract
Emergent Communication (EC) provides a unique window into the language systems that emerge autonomously when agents are trained to jointly achieve shared goals. However, it is difficult to interpret EC and evaluate its relationship with natural languages (NL). This study employs unsupervised neural machine translation (UNMT) techniques to decipher ECs formed during referential games with varying task complexities, influenced by the semantic diversity of the environment. Our findings demonstrate UNMT's potential to translate EC, illustrating that task complexity characterized by semantic diversity enhances EC translatability, while higher task complexity with constrained semantic variability exhibits pragmatic EC, which, although challenging to interpret, remains suitable for translation. This research marks the first attempt, to our knowledge, to translate EC without the aid of parallel data.
Abstract (translated)
新兴通信（EC）为了解当智能体被训练以共同实现共享目标时自主形成的语言系统提供了一个独特的窗口。然而，很难解释EC并评估其与自然语言(NL)的关系。本研究采用无监督神经机器翻译(UNMT)技术来解析在具有不同任务复杂度的指称游戏中形成的EC，这些游戏受到环境语义多样性的影响。我们的研究结果表明，UNMT有潜力将EC进行翻译，并展示了以下发现：由语义多样性定义的任务复杂性增强了EC的可译性；而更高任务复杂性且语义变化受限的情况下，则形成了具有实用性的EC，尽管这种情况下解释起来较为困难，但仍然适合于翻译。据我们所知，这项研究是首次尝试在没有平行数据辅助的情况下将EC进行翻译的研究。
URL
https://arxiv.org/abs/2502.07552
PDF
-
TWICE: What Advantages Can Low-Resource Domain-Specific Embedding Model Bring? - A Case Study on Korea Financial Texts
Abstract
Domain specificity of embedding models is critical for effective performance. However, existing benchmarks, such as FinMTEB, are primarily designed for high-resource languages, leaving low-resource settings, such as Korean, under-explored. Directly translating established English benchmarks often fails to capture the linguistic and cultural nuances present in low-resource domains. In this paper, titled TWICE: What Advantages Can Low-Resource Domain-Specific Embedding Models Bring? A Case Study on Korea Financial Texts, we introduce KorFinMTEB, a novel benchmark for the Korean financial domain, specifically tailored to reflect its unique cultural characteristics in low-resource languages. Our experimental results reveal that while the models perform robustly on a translated version of FinMTEB, their performance on KorFinMTEB uncovers subtle yet critical discrepancies, especially in tasks requiring deeper semantic understanding, that underscore the limitations of direct translation. This discrepancy highlights the necessity of benchmarks that incorporate language-specific idiosyncrasies and cultural nuances. The insights from our study advocate for the development of domain-specific evaluation frameworks that can more accurately assess and drive the progress of embedding models in low-resource settings.
Abstract (translated)
领域特定的嵌入模型对于有效性能至关重要。然而，现有的基准测试（如FinMTEB）主要针对高资源语言设计，这导致像韩语这样的低资源环境被忽视。直接将已建立的英语基准翻译到其他语言往往无法捕捉这些语言中的文化和语言细微差别。在题为《TWICE：低资源领域特定嵌入模型能带来哪些优势？韩国金融文本案例研究》的研究中，我们介绍了KorFinMTEB，这是一个针对韩语金融领域的全新基准测试，特别注重反映其独特的文化特性。实验结果显示，虽然这些模型在翻译版本的FinMTEB上表现出色，但在KorFinMTEB上的表现却揭示了细微但关键的区别，特别是在需要更深层次语义理解的任务中。这种差异突显了将语言特定的独特性及文化细微差纳入基准测试的重要性。 我们研究中的见解倡导开发领域特定的评估框架，以更加准确地衡量和推动低资源环境下的嵌入模型进步。
URL
https://arxiv.org/abs/2502.07131
PDF
-
Multilingual Non-Autoregressive Machine Translation without Knowledge Distillation
2025-02-06 22:16:28Chenyang Huang, Fei Huang, Zaixiang Zheng, Osmar R. Za\"iane, Hao Zhou, Lili MouAbstract
Multilingual neural machine translation (MNMT) aims at using one single model for multiple translation directions. Recent work applies non-autoregressive Transformers to improve the efficiency of MNMT, but requires expensive knowledge distillation (KD) processes. To this end, we propose an M-DAT approach to non-autoregressive multilingual machine translation. Our system leverages the recent advance of the directed acyclic Transformer (DAT), which does not require KD. We further propose a pivot back-translation (PivotBT) approach to improve the generalization to unseen translation directions. Experiments show that our M-DAT achieves state-of-the-art performance in non-autoregressive MNMT.
Abstract (translated)
多语种神经机器翻译（MNMT）的目标是使用单一模型来处理多个语言之间的翻译方向。最近的工作应用了非自回归Transformer以提高MNMT的效率，但需要昂贵的知识蒸馏(KD)过程。为此，我们提出了一种用于非自回归多语种机器翻译的M-DAT方法。我们的系统利用了近期发展的有向无环变压器（DAT），这种方法不需要知识蒸馏。此外，我们还提出了一种枢轴回译（PivotBT）的方法来改进对未见过的翻译方向的一般化能力。实验表明，我们的M-DAT在非自回归MNMT中达到了最先进的性能。 具体来说： - MNMT旨在使用一个单一模型来进行多种语言之间的相互翻译。 - 最近的研究工作利用了非自回归Transformer技术以提高翻译效率，但这种方法需要复杂且计算成本高昂的知识蒸馏过程。 - 我们提出了一种新的方法M-DAT，它基于有向无环变压器（DAT）架构，无需进行知识蒸馏步骤就能实现高效的多语言翻译。 - 此外，我们还引入了枢轴回译技术来增强模型对新出现的、之前未见过的语言配对之间的翻译能力。 - 实验结果表明，我们的方法在非自回归MNMT领域取得了当前最佳的效果。
URL
https://arxiv.org/abs/2502.04537
PDF
-
Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation
Abstract
In this work, we explore how instance-level memorization in the teacher Neural Machine Translation (NMT) model gets inherited by the student model in sequence-level knowledge distillation (SeqKD). We find that despite not directly seeing the original training data, students memorize more than baseline models (models of the same size, trained on the original data) -- 3.4% for exact matches and 57% for extractive memorization -- and show increased hallucination rates. Further, under this SeqKD setting, we also characterize how students behave on specific training data subgroups, such as subgroups with low quality and specific counterfactual memorization (CM) scores, and find that students exhibit amplified denoising on low-quality subgroups. Finally, we propose a modification to SeqKD named Adaptive-SeqKD, which intervenes in SeqKD to reduce memorization and hallucinations. Overall, we recommend caution when applying SeqKD: students inherit both their teachers' superior performance and their fault modes, thereby requiring active monitoring.
Abstract (translated)
在这项工作中，我们探讨了教师神经机器翻译（NMT）模型中的实例级记忆如何通过序列级别的知识蒸馏（SeqKD）传递给学生模型。研究发现，尽管学生模型没有直接接触原始训练数据，但它们的记忆量比基线模型（与之大小相同，在原始数据上进行训练的模型）更多——精确匹配多出3.4%，提取式记忆多出57%——并且表现出更高的幻觉率。此外，在这种SeqKD设置下，我们还描述了学生模型在特定训练数据子集上的行为表现，例如质量低下的子组和具有特定反事实记忆（CM）得分的子组，并发现学生模型对低质量子组显示出更强的去噪效果。最后，我们提出了一种名为自适应SeqKD的SeqKD改进方法，通过干预减少学生的记忆量和幻觉现象。总体而言，我们在应用SeqKD时建议谨慎行事：学生不仅继承了教师的优点，也继承了他们的缺陷模式，因此需要进行积极监控。
URL
https://arxiv.org/abs/2502.01491
PDF
-
Evaluation of NMT-Assisted Grammar Transfer for a Multi-Language Configurable Data-to-Text System
2025-01-27 15:25:26Andreas Madsack, Johanna Heininger, Adela Schneider, Ching-Yi Chen, Christian Eckard, Robert Wei{\ss}graeberAbstract
One approach for multilingual data-to-text generation is to translate grammatical configurations upfront from the source language into each target language. These configurations are then used by a surface realizer and in document planning stages to generate output. In this paper, we describe a rule-based NLG implementation of this approach where the configuration is translated by Neural Machine Translation (NMT) combined with a one-time human review, and introduce a cross-language grammar dependency model to create a multilingual NLG system that generates text from the source data, scaling the generation phase without a human in the loop. Additionally, we introduce a method for human post-editing evaluation on the automatically translated text. Our evaluation on the SportSett:Basketball dataset shows that our NLG system performs well, underlining its grammatical correctness in translation tasks.
Abstract (translated)
一种多语言数据到文本生成的方法是提前将源语言的语法结构翻译成每个目标语言。这些配置随后由表层实现器和文档规划阶段使用，以生成输出内容。在本文中，我们描述了一种基于规则的NLG（自然语言生成）实现方法，在该方法中通过神经机器翻译（NMT）结合一次性人工审查来翻译配置，并引入跨语言语法依赖模型，以此创建一个多语言NLG系统，可从源数据生成文本，并且无需人工干预即可扩展生成阶段。此外，我们还介绍了一种对自动翻译后的文本进行人工后期编辑评估的方法。在SportSett:Basketball数据集上的评估表明，我们的NLG系统表现良好，突出了其在翻译任务中的语法正确性。
URL
https://arxiv.org/abs/2501.16135
PDF
-
Improving Estonian Text Simplification through Pretrained Language Models and Custom Datasets
2025-01-26 18:10:20Eduard Barbu, Meeri-Ly Muru, Sten Marcus MalvaAbstract
This study introduces an approach to Estonian text simplification using two model architectures: a neural machine translation model and a fine-tuned large language model (LLaMA). Given the limited resources for Estonian, we developed a new dataset, the Estonian Simplification Dataset, combining translated data and GPT-4.0-generated simplifications. We benchmarked OpenNMT, a neural machine translation model that frames text simplification as a translation task, and fine-tuned the LLaMA model on our dataset to tailor it specifically for Estonian simplification. Manual evaluations on the test set show that the LLaMA model consistently outperforms OpenNMT in readability, grammaticality, and meaning preservation. These findings underscore the potential of large language models for low-resource languages and provide a basis for further research in Estonian text simplification.
Abstract (translated)
这项研究介绍了一种使用两种模型架构对爱沙尼亚语文本进行简化的方法：神经机器翻译模型和经过微调的大规模语言模型（LLaMA）。鉴于爱沙尼亚语资源有限，我们开发了一个新的数据集——爱沙尼亚文本简化数据集，该数据集结合了翻译数据和GPT-4.0生成的简化文本。我们将OpenNMT作为基准，这是一种将文本简化视为翻译任务的神经机器翻译模型，并在我们的数据集上对LLaMA模型进行微调，使其专门适用于爱沙尼亚语文本简化。手动评估测试集结果显示，与OpenNMT相比，LLaMA模型在可读性、语法正确性和意义保持方面始终表现更优。这些发现强调了大规模语言模型在低资源语言中的潜力，并为进一步研究爱沙尼亚语文本简化提供了基础。
URL
https://arxiv.org/abs/2501.15624
PDF
-
Faster Machine Translation Ensembling with Reinforcement Learning and Competitive Correction
2025-01-25 13:50:18Kritarth Prasad, Mohammadi Zaki, Pratik Singh, Pankaj WasnikAbstract
Ensembling neural machine translation (NMT) models to produce higher-quality translations than the $L$ individual models has been extensively studied. Recent methods typically employ a candidate selection block (CSB) and an encoder-decoder fusion block (FB), requiring inference across \textit{all} candidate models, leading to significant computational overhead, generally $\Omega(L)$. This paper introduces \textbf{SmartGen}, a reinforcement learning (RL)-based strategy that improves the CSB by selecting a small, fixed number of candidates and identifying optimal groups to pass to the fusion block for each input sentence. Furthermore, previously, the CSB and FB were trained independently, leading to suboptimal NMT performance. Our DQN-based \textbf{SmartGen} addresses this by using feedback from the FB block as a reward during training. We also resolve a key issue in earlier methods, where candidates were passed to the FB without modification, by introducing a Competitive Correction Block (CCB). Finally, we validate our approach with extensive experiments on English-Hindi translation tasks in both directions.
Abstract (translated)
将神经机器翻译（NMT）模型进行集成以产生比单独的$L$个模型更高的质量翻译已经被广泛研究。最近的方法通常使用候选选择模块（CSM）和编码器-解码器融合块（FB），这需要在所有候选模型上执行推理，导致了显著的计算开销，一般为$\Omega(L)$级别。本文介绍了基于强化学习（RL）策略的\textbf{SmartGen}方法，通过选取一小部分固定数量的候选模型，并确定最优组合以传递给融合模块来优化CSM。此外，之前的方法中CSM和FB是独立训练的，导致了次优的NMT性能。我们的DQN基线\textbf{SmartGen}通过在训练过程中使用来自FB块的反馈作为奖励解决了这个问题。我们还解决了一个早期方法中的关键问题，即候选模型直接传递给FB而未进行任何修改，为此引入了竞争校正模块（CCB）。最后，我们在英语-印地语翻译任务中进行了广泛的实验验证了我们的方法的有效性，包括双向翻译情况。
URL
https://arxiv.org/abs/2501.15219
PDF
-
Extend Adversarial Policy Against Neural Machine Translation via Unknown Token
2025-01-21 14:43:04Wei Zou, Shujian Huang, Jiajun ChenAbstract
Generating adversarial examples contributes to mainstream neural machine translation~(NMT) robustness. However, popular adversarial policies are apt for fixed tokenization, hindering its efficacy for common character perturbations involving versatile tokenization. Based on existing adversarial generation via reinforcement learning~(RL), we propose the `DexChar policy' that introduces character perturbations for the existing mainstream adversarial policy based on token substitution. Furthermore, we improve the self-supervised matching that provides feedback in RL to cater to the semantic constraints required during training adversaries. Experiments show that our method is compatible with the scenario where baseline adversaries fail, and can generate high-efficiency adversarial examples for analysis and optimization of the system.
Abstract (translated)
生成对抗样本有助于主流神经机器翻译（NMT）的稳健性。然而，流行的对抗策略适用于固定的分词方式，这阻碍了其在涉及多样化分词的常见字符扰动中的有效性。基于现有的通过强化学习（RL）进行对抗生成方法，我们提出了“DexChar 策略”，该策略引入了基于词汇替换的现有主流对抗策略下的字符扰动。此外，我们改进了自我监督匹配以提供训练对手时所需的语义约束方面的反馈。实验表明，我们的方法在基线对抗样本失效的情况下仍然有效，并且能够生成用于分析和优化系统的高效对抗样例。
URL
https://arxiv.org/abs/2501.12183
PDF
-
Proverbs Run in Pairs: Evaluating Proverb Translation Capability of Large Language Model
2025-01-21 07:54:22Minghan Wang, Viet-Thanh Pham, Farhad Moghimifar, Thuy-Trang VuAbstract
Despite achieving remarkable performance, machine translation (MT) research remains underexplored in terms of translating cultural elements in languages, such as idioms, proverbs, and colloquial expressions. This paper investigates the capability of state-of-the-art neural machine translation (NMT) and large language models (LLMs) in translating proverbs, which are deeply rooted in cultural contexts. We construct a translation dataset of standalone proverbs and proverbs in conversation for four language pairs. Our experiments show that the studied models can achieve good translation between languages with similar cultural backgrounds, and LLMs generally outperform NMT models in proverb translation. Furthermore, we find that current automatic evaluation metrics such as BLEU, CHRF++ and COMET are inadequate for reliably assessing the quality of proverb translation, highlighting the need for more culturally aware evaluation metrics.
Abstract (translated)
尽管机器翻译（MT）在性能上取得了显著成就，但在翻译语言中的文化元素方面仍存在不足，例如成语、谚语和口语表达。本文研究了最先进的神经机器翻译（NMT）和大型语言模型（LLMs）在翻译谚语方面的能力，这些谚语深深植根于特定的文化背景之中。我们为四种语言对构建了一个独立谚语和会话语境中的谚语的翻译数据集。我们的实验表明，对于文化背景相似的语言而言，所研究的模型能够实现较好的翻译效果，并且大型语言模型在谚语翻译中通常优于神经机器翻译模型。此外，我们发现现有的自动评估指标（如BLEU、CHRF++和COMET）不足以可靠地衡量谚语翻译的质量，这突显了需要更多具有文化意识的评估标准的重要性。
URL
https://arxiv.org/abs/2501.11953
PDF
-
ViBidirectionMT-Eval: Machine Translation for Vietnamese-Chinese and Vietnamese-Lao language pair
2025-01-15 06:40:26Hong-Viet Tran, Minh-Quy Nguyen, Van-Vinh NguyenAbstract
This paper presents an results of the VLSP 2022-2023 Machine Translation Shared Tasks, focusing on Vietnamese-Chinese and Vietnamese-Lao machine translation. The tasks were organized as part of the 9th, 10th annual workshop on Vietnamese Language and Speech Processing (VLSP 2022, VLSP 2023). The objective of the shared task was to build machine translation systems, specifically targeting Vietnamese-Chinese and Vietnamese-Lao translation (corresponding to 4 translation directions). The submission were evaluated on 1,000 pairs for testing (news and general domains) using established metrics like BLEU [11] and SacreBLEU [12]. Additionally, system outputs also were evaluated with human judgment provided by experts in Chinese and Lao languages. These human assessments played a crucial role in ranking the performance of the machine translation models, ensuring a more comprehensive evaluation.
Abstract (translated)
本文介绍了VLSP 2022-2023机器翻译共享任务的结果，重点在于越南语到中文和老挝文的机器翻译。这些任务作为第九届和第十届越南语言与语音处理年度研讨会（VLSP 2022、VLSP 2023）的一部分而组织进行。共享任务的目标是构建针对越中和越老翻译系统的机器翻译系统（涵盖四个翻译方向）。提交的作品在1,000个测试对（新闻和通用领域）上使用BLEU [11] 和SacreBLEU [12] 等公认的指标进行了评估。此外，系统输出还通过中文和老挝语言专家提供的主观评价进行了评估。这些人类评估对于排名机器翻译模型的性能起到了关键作用，确保了更全面的评估。
URL
https://arxiv.org/abs/2501.08621
PDF
-
AFRIDOC-MT: Document-level MT Corpus for African Languages
2025-01-10 22:49:29Jesujoba O. Alabi, Israel Abebe Azime, Miaoran Zhang, Cristina Espa\~na-Bonet, Rachel Bawden, Dawei Zhu, David Ifeoluwa Adelani, Clement Oyeleke Odoje, Idris Akinade, Iffat Maab, Davis David, Shamsuddeen Hassan Muhammad, Neo Putini, David O. Ademuyiwa, Andrew Caines, Dietrich KlakowAbstract
This paper introduces AFRIDOC-MT, a document-level multi-parallel translation dataset covering English and five African languages: Amharic, Hausa, Swahili, Yorùbá, and Zulu. The dataset comprises 334 health and 271 information technology news documents, all human-translated from English to these languages. We conduct document-level translation benchmark experiments by evaluating neural machine translation (NMT) models and large language models (LLMs) for translations between English and these languages, at both the sentence and pseudo-document levels. These outputs are realigned to form complete documents for evaluation. Our results indicate that NLLB-200 achieved the best average performance among the standard NMT models, while GPT-4o outperformed general-purpose LLMs. Fine-tuning selected models led to substantial performance gains, but models trained on sentences struggled to generalize effectively to longer documents. Furthermore, our analysis reveals that some LLMs exhibit issues such as under-generation, repetition of words or phrases, and off-target translations, especially for African languages.
Abstract (translated)
这篇论文介绍了AFRIDOC-MT，这是一个涵盖英语和五种非洲语言（阿姆哈拉语、豪萨语、斯瓦希里语、约鲁巴语和祖鲁语）的文档级多平行翻译数据集。该数据集包括334份健康类文档和271份信息技术新闻文档，所有这些文档均由人工从英语翻译成上述五种非洲语言。 我们在句子级别和伪文档级别上对神经机器翻译（NMT）模型和大型语言模型（LLM）进行了文档级翻译基准实验，并将输出重新排列以形成完整的文档进行评估。我们的结果显示，在标准的NMT模型中，NLLB-200取得了最佳平均性能，而GPT-4o在通用型LLM中表现更优。对选定模型进行微调可以显著提高性能，但基于句子训练的模型难以有效地泛化到较长文档。 此外，我们的分析还发现了一些LLM存在的问题，例如某些语言生成不足、重复使用单词或短语以及翻译偏离主题的问题，特别是在处理非洲语言时更为明显。
URL
https://arxiv.org/abs/2501.06374
PDF
-
Bridging Dialects: Translating Standard Bangla to Regional Variants Using Neural Models
2025-01-10 06:50:51Md. Arafat Alam Khandaker, Ziyan Shirin Raha, Bidyarthi Paul, Tashreef MuhammadAbstract
The Bangla language includes many regional dialects, adding to its cultural richness. The translation of Bangla Language into regional dialects presents a challenge due to significant variations in vocabulary, pronunciation, and sentence structure across regions like Chittagong, Sylhet, Barishal, Noakhali, and Mymensingh. These dialects, though vital to local identities, lack of representation in technological applications. This study addresses this gap by translating standard Bangla into these dialects using neural machine translation (NMT) models, including BanglaT5, mT5, and mBART50. The work is motivated by the need to preserve linguistic diversity and improve communication among dialect speakers. The models were fine-tuned using the "Vashantor" dataset, containing 32,500 sentences across various dialects, and evaluated through Character Error Rate (CER) and Word Error Rate (WER) metrics. BanglaT5 demonstrated superior performance with a CER of 12.3% and WER of 15.7%, highlighting its effectiveness in capturing dialectal nuances. The outcomes of this research contribute to the development of inclusive language technologies that support regional dialects and promote linguistic diversity.
Abstract (translated)
孟加拉语包括许多地区方言，这增添了其文化丰富性。将标准的孟加拉语翻译成这些地方方言面临着挑战，因为词汇、发音和句子结构在奇蒂格ong（ Chattogram）、锡莱特（Sylhet）、巴里萨尔（Barishal）、瑙阿卡利（Noakhali）和迈门辛吉（Mymensingh）等地区之间存在显著差异。尽管这些方言对于当地身份认同至关重要，但在技术应用中的代表性却不足。本研究通过使用神经机器翻译(NMT)模型——包括BanglaT5、mT5和mBART50——将标准孟加拉语翻译成这些地方方言来填补这一空白。这项工作旨在保护语言多样性并改善讲不同方言的人之间的交流。 模型在“Vashantor”数据集上进行了微调，该数据集中包含来自各种方言的32,500句句子，并通过字符错误率（CER）和单词错误率（WER）进行评估。BanglaT5表现出色，在CER为12.3%和WER为15.7%的情况下，突出显示了它在捕捉方言细微差别的能力方面具有优势。 这项研究的结果有助于开发包容性语言技术，支持地区方言并促进语言多样性。
URL
https://arxiv.org/abs/2501.05749
PDF
-
Quality Estimation based Feedback Training for Improving Pronoun Translation
2025-01-06 13:34:51Harshit Dhankhar, Baban Gain, Asif Ekbal, Yogesh Mani TripathiAbstract
Pronoun translation is a longstanding challenge in neural machine translation (NMT), often requiring inter-sentential context to ensure linguistic accuracy. To address this, we introduce ProNMT, a novel framework designed to enhance pronoun and overall translation quality in context-aware machine translation systems. ProNMT leverages Quality Estimation (QE) models and a unique Pronoun Generation Likelihood-Based Feedback mechanism to iteratively fine-tune pre-trained NMT models without relying on extensive human annotations. The framework combines QE scores with pronoun-specific rewards to guide training, ensuring improved handling of linguistic nuances. Extensive experiments demonstrate significant gains in pronoun translation accuracy and general translation quality across multiple metrics. ProNMT offers an efficient, scalable, and context-aware approach to improving NMT systems, particularly in translating context-dependent elements like pronouns.
Abstract (translated)
代词翻译一直是神经机器翻译（NMT）领域的一个长期挑战，通常需要跨句子的上下文信息来确保语言准确性。为了解决这一问题，我们引入了ProNMT——一个旨在通过利用质量估计(QE)模型和基于生成可能性的独特反馈机制，在不依赖大量人工标注的情况下，迭代地优化预训练NMT模型以提高代词翻译质量和整体翻译准确性的新型框架。该框架结合了QE分数与特定于代词的奖励来指导训练过程，从而更好地处理语言中的细微差别。 广泛实验表明，ProNMT在多项指标上显著提高了代词翻译准确性以及总体翻译质量。这一方法提供了一种高效、可扩展且上下文感知的方式来改进NMT系统，特别是在翻译依赖于上下文的语言元素（如代词）时表现尤为突出。
URL
https://arxiv.org/abs/2501.03008
PDF
-
Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation
2025-01-06 12:42:54Zhi Qu, Yiran Wang, Jiannan Mao, Chenchen Ding, Hideki Tanaka, Masao Utiyama, Taro WatanabeAbstract
The multilingual neural machine translation (MNMT) enables arbitrary translations across multiple languages by training a model with limited parameters using parallel data only. However, the performance of such MNMT models still lags behind that of large language models (LLMs), limiting their practicality. In this work, we address this limitation by introducing registering to achieve the new state-of-the-art of decoder-only MNMT models. Specifically, we insert a set of artificial tokens specifying the target language, called registers, into the input sequence between the source and target tokens. By modifying the attention mask, the target token generation only pays attention to the activation of registers, representing the source tokens in the target language space. Experiments on EC-40, a large-scale benchmark, show that our method outperforms related methods driven by optimizing multilingual representations. We further scale up and collect 9.3 billion sentence pairs across 24 languages from public datasets to pre-train two models, namely MITRE (multilingual translation with registers). One of them, MITRE-913M, outperforms NLLB-3.3B, achieves comparable performance with commercial LLMs, and shows strong adaptability in fine-tuning. Finally, we open-source our models to facilitate further research and development in MNMT: this https URL.
Abstract (translated)
多语言神经机器翻译（MNMT）通过仅使用平行数据训练具有有限参数的模型，实现了任意多个语言之间的翻译。然而，这种MNMT模型的性能仍然落后于大型语言模型（LLMs），这限制了其实际应用价值。在本项工作中，我们通过引入注册机制来解决这一局限性，并达到了解码器独立MNMT模型的新技术水平。具体来说，在输入序列中，我们将一组指定目标语言的人工标记——称为注册——插入到源语言和目标语言令牌之间。通过对注意力掩码进行修改，目标令牌生成仅关注激活的注册标记，这代表了在目标语言空间中的源语言令牌。 我们在EC-40这一大规模基准测试上的实验表明，我们的方法优于基于优化多语言表示的相关方法。我们进一步扩大规模，在公共数据集中收集了跨越24种语言共计93亿个句子对，并使用这些数据预训练了两个模型，即MITRE（带有注册的多语言翻译）。其中的一个模型MITRE-913M超越了NLLB-3.3B，实现了与商业LLMs相当的表现，并在微调中表现出强大的适应性。最后，我们开源我们的模型，以促进MNMT领域的进一步研究和发展：[此链接](请将此处的占位符替换为实际的URL)。
URL
https://arxiv.org/abs/2501.02979
PDF
-
Crossing Language Borders: A Pipeline for Indonesian Manhwa Translation
2025-01-03 04:32:27Nithyasri Narasimhan, Sagarika SinghAbstract
In this project, we develop a practical and efficient solution for automating the Manhwa translation from Indonesian to English. Our approach combines computer vision, text recognition, and natural language processing techniques to streamline the traditionally manual process of Manhwa(Korean comics) translation. The pipeline includes fine-tuned YOLOv5xu for speech bubble detection, Tesseract for OCR and fine-tuned MarianMT for machine translation. By automating these steps, we aim to make Manhwa more accessible to a global audience while saving time and effort compared to manual translation methods. While most Manhwa translation efforts focus on Japanese-to-English, we focus on Indonesian-to-English translation to address the challenges of working with low-resource languages. Our model shows good results at each step and was able to translate from Indonesian to English efficiently.
Abstract (translated)
在这个项目中，我们开发了一种实用且高效的解决方案，用于自动化从印尼语到英语的漫画（又称Manhwa，即韩式漫画）翻译过程。我们的方法结合了计算机视觉、文本识别和自然语言处理技术，以简化传统手动翻译Manhwa的过程。该流程包括使用经过微调的YOLOv5xu进行气泡检测、Tesseract进行OCR文字识别以及经过微调的MarianMT用于机器翻译。通过自动化这些步骤，我们旨在使更多的Manhwa作品能够被全球观众接触和理解，并且相比手动翻译方法可以节省大量时间和精力。 大多数Manhwa翻译工作主要集中在日语到英语的翻译上，而我们的项目则专注于印尼语到英语的翻译，以解决处理低资源语言所面临的挑战。我们的模型在每个步骤中都显示出良好的效果，并能够高效地完成从印尼语到英语的翻译任务。
URL
https://arxiv.org/abs/2501.01629
PDF
-
Sinhala Transliteration: A Comparative Analysis Between Rule-based and Seq2Seq Approaches
2024-12-31 16:27:47Yomal De Mel, Kasun Wickramasinghe, Nisansa de Silva, Surangika RanathungaAbstract
Due to reasons of convenience and lack of tech literacy, transliteration (i.e., Romanizing native scripts instead of using localization tools) is eminently prevalent in the context of low-resource languages such as Sinhala, which have their own writing script. In this study, our focus is on Romanized Sinhala transliteration. We propose two methods to address this problem: Our baseline is a rule-based method, which is then compared against our second method where we approach the transliteration problem as a sequence-to-sequence task akin to the established Neural Machine Translation (NMT) task. For the latter, we propose a Transformer-based Encode-Decoder solution. We witnessed that the Transformer-based method could grab many ad-hoc patterns within the Romanized scripts compared to the rule-based method. The code base associated with this paper is available on GitHub - this https URL
Abstract (translated)
由于便利性和技术知识的缺乏，音译（即用罗马字母拼写本地脚本而非使用本土化工具）在如僧伽罗语这样的资源匮乏语言中非常普遍，尽管这些语言有自己独特的书写系统。在这项研究中，我们的重点是罗马化的僧伽罗语文本转写。我们提出了两种方法来解决这个问题：一种是基于规则的方法作为基准，然后将其与另一种方法进行比较，在这种方法中，我们将音译问题视为类似成熟的神经机器翻译（NMT）任务的序列到序列任务。对于后者，我们提出了一种基于Transformer的编码-解码解决方案。我们发现，相比于基于规则的方法，基于Transformer的方法能够抓住罗马化脚本中的许多临时模式。与这篇论文相关的代码库可在GitHub上获取 - 这个链接：[请在此处插入实际URL]
URL
https://arxiv.org/abs/2501.00529
PDF
