Learning To Focus: Anatomy-Guided Attention Regularization for Medical Image Classification
2026-07-12 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors designed a method called Locus to help medical image classifiers focus on important body parts when making decisions. Instead of needing lots of manual drawings showing the exact shapes of these parts, they use existing powerful models that already know where these parts are. Their method gently encourages the classifier to pay more attention to these important areas and less to irrelevant background. They tested this on different types of medical images and found it improved how well the classifiers worked and where they focused their attention.
Medical image classificationAttention regularizationSegmentation foundation modelsAnatomical shape priorsDermoscopyX-ray imagingHistopathologyCardiac MRISpatial supervisionClassifier attention
Authors
Tonmoy Hossain, Atiqur Rahman, Farhana Hossain Swarnali, Miaomiao Zhang
Abstract
Medical image classification models are ideally expected to identify diagnostically relevant regions while making predictions, yet standard classification losses rarely provide spatial supervision. Explicit supervision via anatomical shape information, such as segmentation masks of task-relevant anatomy, has been shown to guide the network toward regions relevant to the target prediction. However, obtaining such masks incurs substantial manual annotation effort and computational overhead. With the advent of segmentation foundation models that exhibit strong localization of anatomical structures across diverse imaging modalities, we leverage this capability to extract anatomical shape priors without the burden of training a dedicated segmentation model. In this paper, we propose a new framework, Locus, an anatomical attention regularization framework that leverages pretrained segmentation foundation models to guide a classifier's attention toward diagnostically meaningful anatomical structures across diverse imaging modalities. Instead of enforcing pixel-wise alignment with the foundation-model-derived mask, we introduce a regularization term that adaptively balances attention between anatomical (foreground) and background regions, penalizing the classifier when background attention dominates. We validate Locus on eight diverse medical imaging datasets spanning dermoscopy, X-ray, histopathology, and cardiac MRI, showing consistent gains in classification performance alongside improved anatomically grounded attention.