Abstract
While transformer-based models achieve strong performance on text classification, we explore whether masking input tokens can further enhance their effectiveness. We propose token masking regularization, a simple yet theoretically motivated method that randomly replaces input tokens with a special [MASK] token at probability p. This introduces stochastic perturbations during training, leading to implicit gradient averaging that encourages the model to capture deeper inter-token dependencies. Experiments on language identification and sentiment analysis -- across diverse models (mBERT, Qwen2.5-0.5B, TinyLlama-1.1B) -- show consistent improvements over standard regularization techniques. We identify task-specific optimal masking rates, with p = 0.1 as a strong general default. We attribute the gains to two key effects: (1) input perturbation reduces overfitting, and (2) gradient-level smoothing acts as implicit ensembling.
Abstract (translated)
虽然基于变压器的模型在文本分类任务中表现强劲，但我们探索了通过屏蔽输入令牌是否可以进一步提升其效果。我们提出了一种简单且具有理论依据的方法——令牌屏蔽正则化：以概率 p 随机将输入令牌替换为特殊 [MASK] 令牌。这种方法在训练过程中引入随机扰动，导致隐式的梯度平均化，促使模型捕捉更深层次的令牌间依赖关系。 在语言识别和情感分析任务上进行实验（涵盖多种模型，如 mBERT、Qwen2.5-0.5B 和 TinyLlama-1.1B），结果显示该方法相对于标准正则化技术带来了持续性的改进。我们发现特定任务的最佳屏蔽率，并确定 p = 0.1 是一种通用的良好默认值。 我们认为这种提升主要源于两个关键效应：(1) 输入扰动减少了过拟合，以及 (2) 梯度级别的平滑作用等同于隐式的集成学习（即模型通过梯度水平的平滑处理，在训练过程中模拟了不同输入情况下的多种模型行为）。
URL
https://arxiv.org/abs/2505.11746
