Knowledge Distillation for Automated AI Tutor Evaluation
2026-07-12 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created FATE, a special language model designed to check how good AI tutors are at teaching. Because there isn’t much labeled data for this task, they used a technique called knowledge distillation to improve FATE's accuracy by a lot. They tested FATE by rating lessons from popular AI tutors like ChatGPT and found that Gemini 2.5 Flash scored the highest. This work helps automatically judge how well AI tutors teach based on specific teaching skills.
Large Language ModelsAI tutorsPedagogical evaluationKnowledge distillationMistake IdentificationGuidance assessmentInstructional responseBenchmarkingBEA 2025 Shared Task
Authors
Tahmid Al Hannan, Diego Garcia, Alex Njoroge, Suha Al Juboori, Tarek Sakakini
Abstract
The rapid integration of Large Language Models (LLMs) into K-12 and higher education has outpaced the development of reliable methods for evaluating their pedagogical quality. As the research community starts to explore the space of automating evaluation of AI tutors, we introduce FATE (FLC AI Tutor Evaluator), a specialized 8B-parameter language model designed to evaluate AI tutors. Aligned with the four core evaluation tracks from the BEA 2025 Shared Task, our model assesses pedagogical ability across Mistake Identification, Mistake Location, Guidance, and Actionability. Because pedagogical evaluation is a specialized task with limited labeled data, we leverage knowledge distillation from a frontier LLM to generate additional supervision, yielding absolute performance gains up to 22.63 percentage points. Finally, we demonstrate FATE's utility as an automated evaluator by benchmarking instructional responses generated by popular commercial models, including ChatGPT, Claude, Gemini, and DeepSeek. On average, we have found that Gemini 2.5 Flash perfomed best (82.88%), then ChatGPT 5.5 Instant (80.75%), followed by DeepSeek V4 Flash (80.13%) and Claude Sonnet 4.6 (74.00%).