Abstract
Unsupervised cross-lingual transfer involves transferring knowledge between languages without explicit supervision. Although numerous studies have been conducted to improve performance in such tasks by focusing on cross-lingual knowledge, particularly lexical and syntactic knowledge, current approaches are limited as they only incorporate syntactic or lexical information. Since each type of information offers unique advantages and no previous attempts have combined both, we attempt to explore the potential of this approach. In this paper, we present a novel framework called "Lexicon-Syntax Enhanced Multilingual BERT" that combines both lexical and syntactic knowledge. Specifically, we use Multilingual BERT (mBERT) as the base model and employ two techniques to enhance its learning capabilities. The code-switching technique is used to implicitly teach the model lexical alignment information, while a syntactic-based graph attention network is designed to help the model encode syntactic structure. To integrate both types of knowledge, we input code-switched sequences into both the syntactic module and the mBERT base model simultaneously. Our extensive experimental results demonstrate this framework can consistently outperform all baselines of zero-shot cross-lingual transfer, with the gains of 1.0~3.7 points on text classification, named entity recognition (ner), and semantic parsing tasks. Keywords:cross-lingual transfer, lexicon, syntax, code-switching, graph attention network
Abstract (translated)
无监督跨语言转移涉及在没有任何明确监督的情况下在语言之间传递知识。尽管已经进行了大量研究，以通过关注跨语言知识来提高此类任务的性能，特别是词汇和句法知识，但目前的方法仍然有限，因为它们仅包括语义或词汇信息。由于每种信息都具有独特的优势，并且没有 previous 尝试将两种信息相结合，因此我们试图探索这种方法的潜力。在本文中，我们提出了一个名为 "Lexicon-Syntax Enhanced Multilingual BERT" 的新框架，结合了词汇和句法知识。具体来说，我们使用多语言 BERT（mBERT）作为基础模型，并采用两种技术来增强其学习能力。代码转换技术用于含蓄地教导模型词汇对齐信息，而基于句法的图注意力网络旨在帮助模型编码语义结构。为了整合两种知识，我们将代码转换序列同时输入到语义模块和 mBERT 基础模型中。我们进行了广泛的实验研究，结果表明，与其他零散的跨语言转移 baseline 相比，该框架可以始终如一地优于所有基线，在文本分类、命名实体识别（NER）和语义解析任务中的得分增加了 1.0~3.7 点。关键词：跨语言转移，词汇，语法，代码转换，图注意力网络
URL
https://arxiv.org/abs/2404.16627
