Abstract
When a teacher provides examples for a student to study, these examples must be informative, enabling a student to progress from their current state toward a target concept or skill. Good teachers must therefore simultaneously infer what students already know and adapt their teaching to students' changing state of knowledge. There is increasing interest in using computational models, particularly large language models, as pedagogical tools. As students, language models in particular have shown a remarkable ability to adapt to new tasks given small numbers of examples. But how effectively can these models adapt as teachers to students of different types? To study this question, we introduce a suite of models and evaluation methods we call AdapT. AdapT has two components: (1) a collection of simulated Bayesian student models that can be used for evaluation of automated teaching methods; (2) a platform for evaluation with human students, to characterize the real-world effectiveness of these methods. We additionally introduce (3) AToM, a new probabilistic model for adaptive teaching that jointly infers students' past beliefs and optimizes for the correctness of future beliefs. In evaluations of simulated students across three learning domains (fraction arithmetic, English morphology, function learning), AToM systematically outperforms LLM-based and standard Bayesian teaching models. In human experiments, both AToM and LLMs outperform non-adaptive random example selection. Our results highlight both the difficulty of the adaptive teaching task and the potential of learned adaptive models for solving it.
Abstract (translated)
当老师为学生提供学习例子时，这些例子必须是有指导性的，使学生能够从现有状态进展到目标概念或技能。因此，好的老师必须同时推断学生已经知道的内容，并根据学生知识的改变情况进行调整。随着人们对计算模型的兴趣增加，特别是大型语言模型，作为教学工具的应用也越来越受到关注。尤其是大型语言模型，在给定少量示例的情况下表现出惊人的适应能力。但是这些模型作为教师对不同类型的学生有哪些适应能力呢？为了研究这个问题，我们引入了一个名为AdapT的系列模型和评估方法。AdapT有两个组件：（1）一组用于评估自动教学方法的模拟贝叶斯学生模型；（2）一个用于评价人学生模型的平台，以评估这些方法在现实世界中的有效性。此外，我们还引入了（3）AToM，一种新的概率模型，用于适应性教学，它共同推断学生的先验信念并优化未来的信念正确性。在三个学习领域（分数代数，英语形态学，函数学习）的模拟学生评估中，AdapT系统性地优于LLM基和标准贝叶斯教学模型。在人类实验中，AToM和LLM都胜过了非适应性随机示例选择。我们的结果突出了适应性教学任务的困难以及学习适应性模型解决这个问题的潜在可能性。
URL
https://arxiv.org/abs/2405.04495
