MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformers
2026-07-30 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors present MixFrag, a method that smartly assigns different precision levels to parts of Vision Transformers when converting them to use fewer bits, improving efficiency. They measure how sensitive each component is to quantization errors using a special metric and then allocate bits accordingly under a set budget. Testing on various vision tasks shows MixFrag performs better or as well as existing methods, sometimes significantly. Their work shows that understanding component fragility helps optimize bit allocation for mixed-precision quantization.
Post-training quantizationVision TransformersMixed-precisionKullback–Leibler divergenceBit allocationMultiple-Choice Knapsack ProblemImageNet-1KCOCO datasetObject detectionInstance segmentation
Authors
Md. Mehrab Hossain Opi, Robiul Islam Ryad, Md. Umar Faruk
Abstract
Post-training quantization (PTQ) has emerged as an effective solution for deploying Vision Transformers (ViTs) on resource-constrained devices. However, existing PTQ methods typically employ uniform bit-widths across transformer components, overlooking their heterogeneous sensitivity to quantization and leading to inefficient precision allocation. In this paper, we propose {MixFrag, a fragility-guided mixed-precision PTQ framework for Vision Transformers. MixFrag first estimates component-level quantization fragility by measuring the Kullback--Leibler (KL) divergence between full-precision and isolated quantized output distributions using a small calibration set. It then formulates bit allocation as a Multiple-Choice Knapsack Problem (MCKP), enabling adaptive layer-wise precision assignment under a target bit budget. Extensive experiments on ImageNet-1K across multiple Vision Transformer architectures demonstrate that MixFrag achieves competitive classification performance under practical mixed-precision settings. Furthermore, evaluations on COCO object detection and instance segmentation show that MixFrag achieves state-of-the-art performance among existing mixed-precision PTQ methods, improving the previous best method by up to 9.6 AP under the challenging MP3/MP3 setting. Additional analyses validate the proposed fragility metric and demonstrate its strong correlation with the learned bit allocation. These results establish MixFrag as an effective framework for mixed-precision post-training quantization of Vision Transformers.