Abstract
This thesis argues that the currently widely used Natural Language Processing algorithms possibly have various limitations related to the properties of the texts they handle and produce. With the wide adoption of these tools in rapid progress, we must ask what these limitations are and what are the possible implications of integrating such tools even more deeply into our daily lives. As a testbed, we have chosen the task of Neural Machine Translation (NMT). Nevertheless, we aim for general insights and outcomes, applicable even to current Large Language Models (LLMs). We ask whether the algorithms used in NMT have inherent inductive biases that are beneficial for most types of inputs but might harm the processing of untypical texts. To explore this hypothesis, we define a set of measures to quantify text diversity based on its statistical properties, like uniformity or rhythmicity of word-level surprisal, on multiple scales (sentence, discourse, language). We then conduct a series of experiments to investigate whether NMT systems struggle with maintaining the diversity of such texts, potentially reducing the richness of the language generated by these systems, compared to human translators. We search for potential causes of these limitations rooted in training objectives and decoding algorithms. Our ultimate goal is to develop alternatives that do not enforce uniformity in the distribution of statistical properties in the output and that allow for better global planning of the translation, taking into account the intrinsic ambiguity of the translation task.
Abstract (translated)
本论文认为，目前广泛使用的自然语言处理算法可能存在与它们处理和产生的文本相关的各种局限性。随着这些工具的广泛采用，我们必须问一问这些局限性是什么，以及将这样的工具更深入地融入我们的日常生活中可能产生的后果。作为实验台，我们选择了自然机器翻译（NMT）作为任务。然而，我们的目标是获取一般性的见解和结论，这些结论对目前的较大语言模型（LLMs）也适用。我们问NMT中使用的算法是否具有固有的归纳偏见，对大多数类型的输入有益，但对不规则文本的加工造成损害。为了探讨这个假设，我们定义了一系列指标，用于量化基于统计特征文本的多样性，如单词级别的出人意料的统一性或节奏性，在多个尺度（句子、段落、语言）上进行。然后我们进行了一系列实验，以调查NMT系统是否难以维持这种文本多样性，从而减少由这些系统生成的语言的丰富程度，与人类翻译者相比。我们寻找导致这些局限性的潜在原因，源于训练目标和解码算法的目标。我们最终的目标是开发出不需要在输出和分布的统计特性上实现统一性的替代方案，允许更好的全局规划翻译，考虑到翻译任务的固有歧义性。
URL
https://arxiv.org/abs/2409.09568
