Abstract
We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both modeling paradigms within a single transformer stack: GPT-BERT can be transparently used like any standard causal or masked language model. We test the pretraining process that enables this flexible behavior on the BabyLM Challenge 2024. The results show that the hybrid pretraining outperforms masked-only or causal-only models. We openly release the models, training corpora and code.
Abstract (translated)
我们提出了一种将掩码语言模型与因果语言模型相结合的简单方法。这种混合训练目标产生了一个能够在单一变压器堆栈中结合两种建模范式优势的模型：GPT-BERT 可以像任何标准的因果或掩码语言模型一样透明地使用。我们在 BabyLM 挑战 2024 中测试了能够实现这一灵活行为的预训练过程。结果表明，混合预训练优于仅使用掩码或仅使用因果模型的方法。我们公开发布了这些模型、训练语料库和代码。
URL
https://arxiv.org/abs/2410.24159
