A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks
2026-07-02 • Artificial Intelligence
Artificial IntelligenceMachine Learning
AI summaryⓘ
The authors created a small, tough test with real clinical cases to see how well advanced AI models perform in medicine. They found that the models often missed the most important parts of the tasks, doing better on less critical details. Three AI models were tested, and none did well on half of the critical criteria. The authors also showed a reliable way to automatically grade AI answers compared to experts. This work lays the groundwork for building bigger and more accurate medical AI benchmarks.
clinical benchmarksrubric evaluationmedical AIlarge language modelsclinical prioritiesautomated gradingHealthBenchMECE rubricAI performancemedical specialties
Authors
Samiha A. Ismail, Fan X. Chen, Ali Merali
Abstract
Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinical performance is far from solved - its "Hard" subset top score remains 32%. We present a small, deliberately difficult evaluation dataset of five clinician-authored clinical scenarios spanning four specialties (anaesthesia, internal/family medicine, emergency medicine, and obstetrics), each accompanied by an atomic, weighted, MECE rubric (25-62 criteria per task; 184 criteria total) authored from a clinician-drafted golden answer. We evaluate three frontier models: GPT 5.4, Claude Opus 4.7, and Gemini 3.1 Pro. Mean rubric pass rates were 0.47 (Claude), 0.39 (GPT), and 0.37 (Gemini). The central finding is an inversion of clinical priority: the highest-weighted (weight-5, critical) criteria passed at only 32.4-41.7%, while low-stakes weight-1 criteria passed at 80-90%. 56 of 108 critical (weight-5) criteria (52%) were satisfied by no model. Three LLM autoraters reproduced expert met/not-met labels on 92.8-94.7% of 552 graded criteria. We position this as a methods-and-preliminary-findings contribution: the five tasks demonstrate a scalable, defensible pipeline ready to develop into a large-scale benchmark.