Abstract
Although vision-language and large language models (VLM and LLM) offer promising opportunities for AI-driven educational assessment, their effectiveness in real-world classroom settings, particularly in underrepresented educational contexts, remains underexplored. In this study, we evaluated the performance of a state-of-the-art VLM and several LLMs on 646 handwritten exam responses from grade 4 students in six Indonesian schools, covering two subjects: Mathematics and English. These sheets contain more than 14K student answers that span multiple choice, short answer, and essay questions. Assessment tasks include grading these responses and generating personalized feedback. Our findings show that the VLM often struggles to accurately recognize student handwriting, leading to error propagation in downstream LLM grading. Nevertheless, LLM-generated feedback retains some utility, even when derived from imperfect input, although limitations in personalization and contextual relevance persist.
Abstract (translated)
尽管视觉-语言模型（VLM）和大型语言模型（LLM）为基于AI的教育评估提供了许多机会，但它们在现实世界课堂环境中的有效性，尤其是在教育资源不足的情境下，仍然有待深入研究。在这项研究中，我们评估了一种最先进的VLM以及几种LLM在印度尼西亚六所学校的646份四年级学生手写考试答案上的表现，涵盖数学和英语两门学科。这些答卷包含超过1.4万个学生的答题内容，包括多项选择题、简答题和作文题。评估任务包括对这些回答进行评分并生成个性化反馈。我们的研究发现表明，VLM在准确识别学生手写方面经常遇到困难，这会导致下游LLM评分级别的错误传播。尽管如此，在基于不完美的输入数据时，由LLM生成的反馈仍具有一定的实用性，但个性化和情境相关性的局限性仍然存在。
URL
https://arxiv.org/abs/2506.04822
