Abstract
This study aims at improving the performance of scoring student responses in science education automatically. BERT-based language models have shown significant superiority over traditional NLP models in various language-related tasks. However, science writing of students, including argumentation and explanation, is domain-specific. In addition, the language used by students is different from the language in journals and Wikipedia, which are training sources of BERT and its existing variants. All these suggest that a domain-specific model pre-trained using science education data may improve model performance. However, the ideal type of data to contextualize pre-trained language model and improve the performance in automatically scoring student written responses remains unclear. Therefore, we employ different data in this study to contextualize both BERT and SciBERT models and compare their performance on automatic scoring of assessment tasks for scientific argumentation. We use three datasets to pre-train the model: 1) journal articles in science education, 2) a large dataset of students' written responses (sample size over 50,000), and 3) a small dataset of students' written responses of scientific argumentation tasks. Our experimental results show that in-domain training corpora constructed from science questions and responses improve language model performance on a wide variety of downstream tasks. Our study confirms the effectiveness of continual pre-training on domain-specific data in the education domain and demonstrates a generalizable strategy for automating science education tasks with high accuracy. We plan to release our data and SciEdBERT models for public use and community engagement.
Abstract (translated)
本研究旨在自动提高科学教育中对学生响应评分的表现。基于BERT的语言模型在多种语言相关任务中表现出显著优越性。然而,学生科学写作,包括辩论和解释,是特定领域的写作。此外,学生使用的语言与期刊和维基百科等BERT和其现有变体的训练数据不同。这些都表明,使用科学教育数据预先训练特定领域模型可能会提高模型表现。然而,找到合适的数据类型以 contextualize 预训练的语言模型并提高在自动评分学生手写响应方面的表现仍然是未知的。因此,在本研究中,我们使用不同的数据来 contextualize BERT 和 SciBERT 模型,并比较它们在科学辩论评估任务自动评分方面的表现。我们使用三个数据集来预训练模型:1)科学教育期刊文章,2)大规模的学生手写响应数据集(样本大小超过50,000),3)科学辩论任务学生手写响应的小数据集。我们的实验结果显示,从科学问题和响应中构建的特定领域训练集可以提高语言模型在多种后续任务中的表现。我们的研究确认了在教育领域中持续预训练特定数据集的效果,并展示了自动化科学教育任务并以高精度自动化的策略。我们计划将我们的数据和SciEdBERT模型公之于众并进行社区参与。
URL
https://arxiv.org/abs/2301.12031
