L2-Bench: An Evaluation Benchmark for Measuring LLM Capabilities in Second Language Education
2026-07-09 • Computers and Society
Computers and Society
AI summaryⓘ
The authors created L2-Bench, a tool to better evaluate AI systems designed for learning second languages, focusing on how well these systems apply teaching methods rather than just knowing facts. They developed a detailed checklist of skills validated by language experts and a way to score AI answers using rubrics. Testing several AI models, they found that one called Claude Opus 4.7 performed best overall, although performance dropped on harder tasks. This benchmark helps educators understand AI strengths and weaknesses for language learning and supports better decisions about using AI in education.
AI-powered educational systemssecond language educationbenchmarkinglarge language modelslearning experience designrubric evaluationcompetency taxonomymodel performancetask-response pairs
Authors
James Edgell, Wm. Matthew Kennedy, Ben Knight, Danielle Carvalho, Martin Ku
Abstract
Despite rapid AI adoption in education, rigorous evaluation of AI-powered educational (AIED) systems remains critically underdeveloped, particularly in second language (L2) education, one of the most common yet least evaluated AI applications. We introduce L2-Bench, an open-source benchmark of 1,000+ task-response pairs to aid the pedagogy-led evaluation of LLM capabilities relating to language learning and assessment. Crucially, L2-Bench measures model performativity on the application of learning experience design principles rather than mere knowledge of those principles or broad learning outcomes. Our contributions include: (1) a validated taxonomy of 12 competencies and 31 subcompetencies validated by 200+ expert practitioners (task authenticity: 4.42/5.00, criteria adequacy: 4.18/5.00); (2) a rubric-based evaluation methodology that we believe can, if adapted, generalize to similar (open-ended, qualitative) disciplines; (3) an evaluation dataset that produces reliable signal about model strengths, weaknesses, and contextual robustness across diverse L2 education scenarios. We find that, among large models, Claude Opus 4.7 performs best overall (85.5%), though is marginally outperformed on several constituent tasks. We also find that performance drops notably on harder tasks (69.9% to 73.4%). L2-Bench provides education stakeholders better methods to make more informed decisions about real-world AIED adoption, use, and governance, while advancing the maturing science of AI evaluations for education.