Abstract
For crosslingual conversation and trade, Neural Machine Translation (NMT) is pivotal yet faces persistent challenges with monotony and repetition in generated content. Traditional solutions that rely on penalizing text redundancy or token reoccurrence have shown limited efficacy, particularly for lengthy article and e-commerce descriptions with inherent redundancy, even with the advent of Large Language Models (LLMs). This paper investigates the underlying causes of textual repetition through the lens of information entropy, attributing the phenomenon to the elevated uncertainty within the input text. To address this, a novel algorithm named Contrastive Token Learning with Similarity Decay (CTSD) is introduced, which modulates the suppression of tokens dynamically, informed by varying attention weights and inter-token distances. Furthermore, an e-commerce dataset comprised of title texts of online real items is compiled and released susceptible to hallucination translations to benchmark the algorithm. Extensive evaluations demonstrate that CTSD significantly outperforms existing approaches in precision and generalizability. Additional online A/B testing underscores its practical value, showing marked improvements in user engagement and conversion. Notably, this method has been implemented with full traffic on eight multilingual sites of this http URL, the largest B2B e-commerce platform in the world.
Abstract (translated)
为了进行跨语言对话和贸易，神经机器翻译（NMT）在生成内容的单调化和重复性方面具有关键作用，但生成的内容中存在持续的挑战。传统解决方案依赖于惩罚文本冗余或词项再出现，对于具有固有冗余的长篇文章和电子商务描述，效果有限，即使在大型语言模型（LLMs）出现的情况下。本文通过信息熵的视角研究了文本重复的潜在原因，将这种现象归因于输入文本中的不确定性增加。为解决这个问题，提出了一种名为Contrastive Token Learning with Similarity Decay（CTSD）的新算法，该算法动态地调节词项的抑制，受到不同注意权重和词项间距离的影响。此外，还编译和发布了一个电子商务数据集，包括在线实物的标题文本，以评估算法的性能。大量评估结果表明，CTSD在精度和泛化能力方面显著优于现有方法。进一步的在线A/B测试证实了其实用价值，表明在用户参与度和转化率方面取得了显著的改善。值得注意的是，这种方法在八个多语言网站的全面流量下实现，这是世界上最大的B2B电子商务平台。
URL
https://arxiv.org/abs/2409.19877
