Abstract
Robot systems in education can leverage Large language models' (LLMs) natural language understanding capabilities to provide assistance and facilitate learning. This paper proposes a multimodal interactive robot (PhysicsAssistant) built on YOLOv8 object detection, cameras, speech recognition, and chatbot using LLM to provide assistance to students' physics labs. We conduct a user study on ten 8th-grade students to empirically evaluate the performance of PhysicsAssistant with a human expert. The Expert rates the assistants' responses to student queries on a 0-4 scale based on Bloom's taxonomy to provide educational support. We have compared the performance of PhysicsAssistant (YOLOv8+GPT-3.5-turbo) with GPT-4 and found that the human expert rating of both systems for factual understanding is the same. However, the rating of GPT-4 for conceptual and procedural knowledge (3 and 3.2 vs 2.2 and 2.6, respectively) is significantly higher than PhysicsAssistant (p < 0.05). However, the response time of GPT-4 is significantly higher than PhysicsAssistant (3.54 vs 1.64 sec, p < 0.05). Hence, despite the relatively lower response quality of PhysicsAssistant than GPT-4, it has shown potential for being used as a real-time lab assistant to provide timely responses and can offload teachers' labor to assist with repetitive tasks. To the best of our knowledge, this is the first attempt to build such an interactive multimodal robotic assistant for K-12 science (physics) education.
Abstract (translated)
机器人系统在教育领域可以利用大型语言模型的自然语言理解能力来提供帮助和促进学习。本文提出了一种基于YOLOv8目标检测、相机、语音识别和聊天机器人使用大型语言模型的多模态交互机器人（PhysicsAssistant），以帮助学生进行物理实验室。我们对十名8年级学生进行用户研究，以实证评估PhysicsAssistant与人类专家的性能。根据布卢姆的分类法，专家根据学生问题的回答给0-4评分，以提供教育支持。我们比较了PhysicsAssistant（YOLOv8+GPT-3.5-turbo）与GPT-4的性能，发现两个系统的 factual understanding 方面的专家评分相同。然而，GPT-4的形而上学和程序知识评分（3.2和3.5， respectively）远高于PhysicsAssistant（2.2和2.6， respectively）。因此，尽管PhysicsAssistant的响应质量相对于GPT-4较低，但它已经展示了在实时实验室助理中提供及时响应并减轻教师劳动力的潜力。据我们所知，这是第一个为K-12科学（物理）教育构建的具有多模态的交互式机器人助手。
URL
https://arxiv.org/abs/2403.18721
