AI summaryⓘ
The authors found that while humans and computer vision models can be equally good at recognizing images, they make different kinds of errors in specific ways. By analyzing these directional mistakes, they identified unique patterns that accuracy alone cannot show. Using a special mathematical framework called Rate-Distortion, they linked these patterns to geometric features and found that humans have many mild differences in recognition, but models have fewer, stronger ones. Training models to be more robust changes these patterns but doesn’t make them more human-like. Their work suggests that studying the direction of errors can reveal how different systems think and generalize under changing conditions.
classification accuracyinductive biasconfusion matrixdirectional confusionRate-Distortion frameworkperturbationrobustness traininggeneralization geometrydeep vision modelsnatural-image categorization
Authors
Leyla Roksan Caglar, Pedro A. M. Mediano, Baihan Lin
Abstract
Humans and modern vision models can reach similar classification accuracy while making systematically different kinds of mistakes - differing not in how often they err, but in who gets mistaken for whom, and in which direction. We show that these directional confusions reveal distinct inductive biases that are invisible to accuracy alone. Using matched human and deep vision model responses on a natural-image categorization task under 12 perturbation types, we quantify asymmetry in confusion matrices and link it to generalization geometry through a Rate-Distortion (RD) framework, summarized by three geometric signatures (slope (beta), curvature (kappa)) and efficiency (AUC). We find that humans exhibit broad but weak asymmetries, whereas deep vision models show sparser, stronger directional collapses. Robustness training reduces global asymmetry but fails to recover the human-like breadth-strength profile of graded similarity. Mechanistic simulations further show that different asymmetry organizations shift the RD frontier in opposite directions, even when matched for performance. Together, these results position directional confusions and RD geometry as compact, interpretable signatures of inductive bias under distribution shift.