MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents
2026-07-03 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created a new test called MedCalc-Pro to better evaluate how well large language models (LLMs) handle medical calculations. Unlike previous tests, MedCalc-Pro includes more complex cases where multiple calculations and unclear instructions are involved, reflecting real-life medical situations. They also developed a new system that can choose and use multiple medical calculators correctly, improving the accuracy of the results. Their tests show this new system works better than existing ones on all levels of difficulty.
large language modelsmedical calculationbenchmarkmulti-tool selectionnested calculationclinical decision supportparameter error propagationmodel evaluationmedical calculatorsevidence review
Authors
Siran Zhao, Ruihui Hou, Ziyue Huai, Chennuo Zhang, Tong Ruan
Abstract
Current benchmarks for evaluating large language models (LLMs) in medical calculation are largely based on simplified settings, where each patient case corresponds to a single calculator and the required tool is explicitly specified in the query. However, real clinical scenarios often require multiple calculators for joint evaluation, nested-scale calculation, and fuzzy queries that do not directly specify the target calculator. To this end, we propose a new medical calculation benchmark, MedCalc-Pro, which covers three progressively challenging task settings: single-calculator, multi-calculator, and nested-calculator calculation settings. MedCalc-Pro contains 2,268 real-world clinical cases, covering 77 medical calculators across 14 clinical departments. Meanwhile, to address the limited performance of existing frameworks and methods in complex clinical scenarios, we further propose a more generalizable agent framework that supports multi-tool selection and nested-tool calling, while suppressing parameter error propagation through structured validation and evidence review. We conduct systematic comparisons across open-source, closed-source, and medical-specialized LLMs, and the results show that our framework achieves the best performance across all three task settings. This work provides a new benchmark and method for evaluating and applying LLMs in challenging medical calculation scenarios.