Abstract
The rapid development of Generative AI is bringing innovative changes to education and assessment. As the prevalence of students utilizing AI for assignments increases, concerns regarding academic integrity and the validity of assessments are growing. This study utilizes the Earth Science I section of the 2025 Korean College Scholastic Ability Test (CSAT) to deeply analyze the multimodal scientific reasoning capabilities and cognitive limitations of state-of-the-art Large Language Models (LLMs), including GPT-4o, Gemini 2.5 Flash, and Gemini 2.5 Pro. Three experimental conditions (full-page input, individual item input, and optimized multimodal input) were designed to evaluate model performance across different data structures. Quantitative results indicated that unstructured inputs led to significant performance degradation due to segmentation and Optical Character Recognition (OCR) failures. Even under optimized conditions, models exhibited fundamental reasoning flaws. Qualitative analysis revealed that "Perception Errors" were dominant, highlighting a "Perception-Cognition Gap" where models failed to interpret symbolic meanings in schematic diagrams despite recognizing visual data. Furthermore, models demonstrated a "Calculation-Conceptualization Discrepancy," successfully performing calculations while failing to apply the underlying scientific concepts, and "Process Hallucination," where models skipped visual verification in favor of plausible but unfounded background knowledge. Addressing the challenge of unauthorized AI use in coursework, this study provides actionable cues for designing "AI-resistant questions" that target these specific cognitive vulnerabilities. By exploiting AI's weaknesses, such as the gap between perception and cognition, educators can distinguish genuine student competency from AI-generated responses, thereby ensuring assessment fairness.
Abstract (translated)
生成式人工智能的迅速发展正在为教育和评估带来创新性的变化。随着学生利用AI完成作业的情况日益增多，关于学术诚信和评估有效性的担忧也在增长。这项研究使用了2025年韩国大学修学能力考试（CSAT）中的地球科学第一部分，对包括GPT-4o、Gemini 2.5 Flash 和 Gemini 2.5 Pro在内的最先进大型语言模型（LLMs）的多模态科学推理能力和认知限制进行了深入分析。设计了三种实验条件（全页输入、单项输入和优化后的多模态输入），以评估不同数据结构下模型的表现。定量结果显示，无结构化的输入导致了由于分段处理和光学字符识别失败而导致的重大性能下降。即使在最优化条件下，模型也展示出基本的推理缺陷。定性分析表明，“感知错误”占据主导地位，这揭示了一个“感知-认知差距”，即尽管能够识别视觉数据，但这些模型却未能解读图表中的象征意义。此外，模型还展示了“计算-概念化不一致”的问题，在这种情况下，虽然成功地完成了计算任务，但却无法应用其中的科学原理，并且出现“过程幻觉”，即在缺乏明确依据的情况下利用看似合理的背景知识而跳过视觉验证步骤。针对未经授权使用AI进行课程作业的问题，这项研究为设计能够应对这些特定认知弱点的“抗AI问题”提供了实际建议。通过利用AI的弱点（例如感知与认知之间的差距），教育工作者可以区分出学生的真实能力与AI生成的答案，从而确保评估的公平性。
URL
https://arxiv.org/abs/2512.15298
