The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally
2026-09-01 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors studied how reducing the precision of large language models (LLMs) affects their accuracy and where errors happen in the models. They tested whether the damage from quantizing weights occurs in specific parts like task-related layers or is linked to weight patterns but found no clear pattern. Instead, fixing accuracy loss requires spreading extra precision across many layers rather than focusing on a few. They also found that improving quantization evenly works better than selectively repairing certain layers for most models. Overall, they suggest testing changes directly rather than guessing where accuracy improvements happen.
post-training quantizationlarge language modelsmixed-precisionmodel accuracycausal interventionquantization damageweight statisticsbit precisionGPTQAWQ
Authors
Jundong Hu, Shekar Ramachandran
Abstract
Post-training quantization (PTQ) is widely used to reduce the cost of serving large language models (LLMs), but its accuracy cost is uneven and is often tuned per model. We study where quantization damage occurs and how to allocate a small additional precision budget. Using causal mixed-precision intervention as ground truth (raise each layer to 8-bit in turn and measure the accuracy it recovers) across 9 open-weight models in 4 architecture families, we test 3 intuitive hypotheses: that quantization damage lives in task circuits, where the model computes, or in weight statistics. None of them predicts which layers benefit from restored precision. Recovery is instead diffuse: for 8 of 9 models, recovering 75% of the gap takes roughly half the layers; the lone exception, Qwen3-8B, is sharply concentrated. At a matched precision budget, spending it globally on finer quantization granularity beats locally repairing the most recoverable layers for all 8 group-128-compatible models (all but OpenLLaMA, whose width rules out group-128), by 21-52 points, including the concentrated Qwen3-8B. We report 2 secondary findings: the residual is budget-limited (8-bit is near-lossless in our evaluation across RTN, GPTQ, and AWQ), and the location of peak recovery correlates with architecture within a family, though not across families. Within this budget setting, global granularity is a better default than selectively protecting critical layers. More broadly, cheap signals that correlate with quantization damage do not necessarily identify where restoring precision improves accuracy; this must be tested with causal intervention.