Testing LLM Arithmetic Reasoning Generalization with Automatic Numeric-Remapping Attacks
2026-06-02 • Cryptography and Security
Cryptography and SecurityArtificial Intelligence
AI summaryⓘ
The authors studied how well large language models (LLMs) handle small changes in numbers within arithmetic word problems without using external tools. They created a new automatic method to change numbers in problems while keeping the original reasoning steps intact, then tested several LLMs on these modified problems. Their experiments found that some models struggled more with number changes on a complex dataset (GSM8K), but did better on simpler, more regular datasets (MAWPS and MultiArith). This suggests that how stable a model is depends on the type of dataset and problem structure. The authors highlight that brittleness to small numeric changes is still a challenge for certain settings.
large language modelsarithmetic word problemsnumerical variationnumeric-remapping attacksreasoning robustnessGSM8KMAWPSMultiArithbenchmark evaluationmodel brittleness
Authors
Malia Barker, Bishal Lakha, Edoardo Serra, Francesco Gullo
Abstract
Large language models achieve strong performance on arithmetic reasoning benchmarks, and one common response to arithmetic brittleness is to delegate computation to code. Yet models are still often used in settings where they must reason directly from natural language, and trustworthy models should solve small-number arithmetic word problems without external tools. Prior work shows that LLMs are sensitive to numerical variation: a model may solve an original problem but fail on structurally similar variants requiring the same reasoning procedure with different numbers. We ask whether this fragility persists under a stricter setting involving small, schema-preserving numeric changes that retain the original reasoning program and avoid large-number stress tests. We introduce an automatic algorithm for generating numeric-remapping attacks on arithmetic word problems. Unlike template-based perturbation methods requiring manual schemas or constraints, our approach derives problem-specific symbolic representations, generates constrained numeric remappings, recomputes gold answers, and realizes transformed questions through deterministic edits guided by LLM-generated edit plans. Stage-wise validation and a high-confidence audit retain reliable attacks, making the pipeline scalable with limited human intervention. We evaluate DeepSeek-R1 (70B), Gemma4 (31B), and GPT-OSS (120B) on GSM8K, MAWPS, and MultiArith. On GSM8K, completed runs show conditional accuracy drops of 12.16 to 25.82 percentage points. MAWPS and MultiArith are far more stable, with most attacked accuracies near or above 98%. These results show that numeric-remapping robustness depends strongly on dataset structure: GSM8K remains sensitive even when reasoning programs are preserved and answers are recomputed, while shorter, more regular datasets are more robust.