MRI-Eval: A Tiered Benchmark for Evaluating LLM Performance on MRI Physics and GE Scanner Operations Knowledge
2026-05-06 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created MRI-Eval, a new test to check how well AI models understand MRI physics and the operations of GE MRI scanners, which are important for research. They found that while AI models do very well on multiple-choice questions (over 90% correct), their performance drops a lot when questions are asked without answer choices, especially for GE scanner operations. This means these models may know facts when given options but struggle to recall detailed technical knowledge freely. The authors suggest MRI-Eval is best used to compare models rather than to claim absolute mastery, and caution should be used when relying on AI for specific MRI scanner protocols.
MRI physicsGE scanner operationslarge language modelsmultiple-choice questionsfree-text recallbenchmarkingMRI-EvalLLM evaluationstem-only questionsvendor-specific knowledge
Authors
Perry E. Radau
Abstract
Background: Existing MRI LLM benchmarks rely mainly on review-book multiple-choice questions, where top proprietary models already score highly, limiting discrimination. No systematic benchmark has evaluated vendor-specific scanner operational knowledge central to research MRI practice. Purpose: We developed MRI-Eval, a tiered benchmark for relative model comparison on MRI physics and GE scanner operations knowledge using primary multiple-choice questions (MCQ), with stem-only and primed diagnostic conditions as complementary analyses. Methods: MRI-Eval includes 1365 scored items across nine categories and three difficulty tiers from textbooks, GE scanner manuals, programming course materials, and expert-generated questions. Five model families were evaluated (GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6, Gemini 2.5 Pro, Llama 3.3 70B). MCQ was primary; stem-only removed options and used an independent LLM judge; primed stem-only tested responses to incorrect user claims. Results: Overall MCQ accuracy was 93.2% to 97.1%. GE scanner operations was the lowest category for every model (88.2% to 94.6%). In stem-only, frontier-model accuracy fell to 58.4% to 61.1%, and Llama 3.3 70B fell to 37.1%; GE scanner operations stem-only accuracy was 13.8% to 29.8%. Conclusion: High MCQ performance can mask weak free-text recall, especially for vendor-specific operational knowledge. MRI-Eval is most informative as a relative comparison benchmark rather than an absolute competency measure and supports caution in using raw LLM outputs for GE-specific protocol guidance.