Abstract
Masked language modeling has become a widely adopted unsupervised technique to pre-train language models. However, the process of selecting tokens for masking is random, and the percentage of masked tokens is typically fixed for the entire training process. In this paper, we propose to adjust the masking ratio and to decide which tokens to mask based on a novel task-informed anti-curriculum learning scheme. First, we harness task-specific knowledge about useful and harmful tokens in order to determine which tokens to mask. Second, we propose a cyclic decaying masking ratio, which corresponds to an anti-curriculum schedule (from hard to easy). We exemplify our novel task-informed anti-curriculum by masking (TIACBM) approach across three diverse downstream tasks: sentiment analysis, text classification by topic, and authorship attribution. Our findings suggest that TIACBM enhances the ability of the model to focus on key task-relevant features, contributing to statistically significant performance gains across tasks. We release our code at this https URL.
Abstract (translated)
遮罩语言模型已成为预训练语言模型中广泛采用的无监督技术。然而，选择进行遮蔽的标记的过程是随机的，并且在整个训练过程中被遮蔽标记的比例通常是固定的。在本文中，我们提出调整遮蔽比例并根据一种新的任务相关反课程学习方案来决定要遮蔽哪些标记。首先，我们利用特定于任务的知识来确定哪些标记是有用的或有害的，从而决定要遮蔽哪些标记。其次，我们提出了一个周期性衰减的遮蔽比率，这对应于从难到易的反课程安排。 我们通过三种不同的下游任务展示了我们的新型任务相关反课程方法：情感分析、按主题分类和作者归属。实验结果表明，TIACBM（Task-Informed Anti-Curriculum Based Masking）能够增强模型聚焦于关键的任务相关信息的能力，在各个任务上带来了统计显著的性能提升。我们在提供的链接中发布了我们的代码。 原文链接： [此处应插入实际的URL](this https URL)
URL
https://arxiv.org/abs/2502.12953
