Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach
2026-07-02 • Artificial Intelligence
Artificial IntelligenceComputation and LanguageComputers and Society
AI summaryⓘ
The authors studied how well four large language models (GPT, Claude Opus, Gemini, and GLM) can grade Linux command-line exam answers compared to human experts. They used a scale with four levels of difficulty, from simple tasks like finding information to complex system management. They found that the best model, Gemini 3.0 Pro with detailed instructions, agreed most closely with human graders, especially on easier questions. The study shows that grading becomes harder for AI as question difficulty rises, and well-structured prompts help improve AI grading accuracy. The authors also created a clear way to decide which questions AI can grade reliably and which need human oversight.
Large Language ModelsCommand-line gradingLinux/bashCognitive taxonomyPrompt engineeringInter-rater agreementPartial creditAutogradingOperational impactMAE (Mean Absolute Error)
Authors
Manuel Alonso-Carracedo, Ruben Fernandez-Boullon, Pedro Celard, Francisco J. Rodriguez-Martinez, Lorena Otero-Cerdeira
Abstract
Scalable and reliable grading of command-line examinations remains a challenge in computing education, where rising enrolments make manual marking difficult and rule-based autograders cannot handle partial credit, equivalent solutions, or syntactic variation. This paper evaluates whether four frontier Large Language Models (GPT, Claude Opus, Gemini, and GLM) can approximate expert judgment when grading short Linux/bash command responses. The study adopts a four-level cognitive taxonomy that combines cognitive complexity and operational impact, ranging from information retrieval (L1) and basic file manipulation (L2) to structural operations (L3) and advanced system management (L4). The models were tested with two prompt variants, a minimal baseline and a rubric-enhanced version, on 1200 real responses from second-year Computer Engineering students independently graded by three expert instructors. Gemini~3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement (ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = -0.014). Agreement declined consistently as taxonomy level increased, with the largest discrepancies at higher levels. Across all models, rubric quality had a larger effect than provider choice, with structured prompts consistently improving agreement. These results show that question complexity is a reliable predictor of the difficulty LLMs face in grading accurately, and they establish a principled, taxonomy-based framework for determining which questions are suitable for AI-assisted grading and which require human review, while also providing a transferable evaluation protocol and prompt templates.