Abstract
An ongoing challenge in current natural language processing is how its major advancements tend to disproportionately favor resource-rich languages, leaving a significant number of under-resourced languages behind. Due to the lack of resources required to train and evaluate models, most modern language technologies are either nonexistent or unreliable to process endangered, local, and non-standardized languages. Optical character recognition (OCR) is often used to convert endangered language documents into machine-readable data. However, such OCR output is typically noisy, and most word alignment models are not built to work under such noisy conditions. In this work, we study the existing word-level alignment models under noisy settings and aim to make them more robust to noisy data. Our noise simulation and structural biasing method, tested on multiple language pairs, manages to reduce the alignment error rate on a state-of-the-art neural-based alignment model up to 59.6%.
Abstract (translated)
当前自然语言处理面临的一个持续挑战是如何过分地倾向于资源丰富的语言,导致许多缺乏资源的语言落后。由于训练和评估模型所需的资源不足,大多数现代语言技术 either不存在或可靠性不高,以处理濒危、当地和未标准化的语言。光学字符识别(OCR)通常用于将濒危语言文档转换为可读数据。然而,这种OCR输出通常很嘈杂,而且大多数词对齐模型不是为了在嘈杂条件下工作而构建的。在这个研究中,我们研究了在嘈杂条件下存在的词级对齐模型,并旨在使其更加稳健地应对嘈杂数据。我们对噪声模拟和结构偏置方法进行了多个语言对的测试,成功地将最先进的神经网络对齐模型的词对齐错误率降低到59.6%。
URL
https://arxiv.org/abs/2301.09685
