Abstract
Zero-Shot Cross-lingual Transfer (ZS-XLT) utilizes a model trained in a source language to make predictions in another language, often with a performance loss. To alleviate this, additional improvements can be achieved through subsequent adaptation using examples in the target language. In this paper, we exploit In-Context Tuning (ICT) for One-Shot Cross-lingual transfer in the classification task by introducing In-Context Cross-lingual Transfer (IC-XLT). The novel concept involves training a model to learn from context examples and subsequently adapting it during inference to a target language by prepending a One-Shot context demonstration in that language. Our results show that IC-XLT successfully leverages target-language examples to improve the cross-lingual capabilities of the evaluated mT5 model, outperforming prompt-based models in the Zero and Few-shot scenarios adapted through fine-tuning. Moreover, we show that when source-language data is limited, the fine-tuning framework employed for IC-XLT performs comparably to prompt-based fine-tuning with significantly more training data in the source language.
Abstract (translated)
Zero-Shot Cross-Lingual Transfer（ZS-XLT）利用在源语言中训练的模型进行预测，往往性能会降低。为缓解这一问题，可以通过使用目标语言中的示例进行后续适应来获得进一步的改进。在本文中，我们利用In-Context Tuning（ICT）为One-Shot Cross-Lingual Transfer（IC-XLT）在分类任务中进行改进。新概念涉及通过在目标语言中准备一个One-Shot上下文演示来训练模型，然后在推理时将其适应目标语言。我们的结果表明，ICT成功利用目标语言示例来提高评估mT5模型的跨语言能力，在零和少样本场景下优于基于提示的模型。此外，我们还证明了当源语言数据有限时，用于ICT的微调框架与通过大量在源语言中的训练数据进行微调的提示方法的表现相当。
URL
https://arxiv.org/abs/2404.02452
