Automated Background Swapping for Robustness against Spurious Backgrounds
2026-06-30 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors address a common problem in image classification where models mistakenly focus on backgrounds that are not truly related to the object being identified. They propose AutoBackSwap, a method that separates the main object (foreground) from the background, creates new backgrounds by filling in missing parts, and mixes different foregrounds and backgrounds to create new training images. This helps the model learn to focus on the object itself rather than misleading background cues. Their approach works well even when the training data never shows the object with different backgrounds, outperforming previous methods in this area.
Deep Neural NetworksImage ClassificationSpurious CorrelationsForeground-Background DisentanglementData AugmentationInpaintingPatch-wise LabelingGeneralizationAutomated Background Swapping
Authors
Cesar Roder, Kajetan Schweighofer
Abstract
Classifiers based on Deep Neural Networks exhibit strong performance across domains, yet can fail catastrophically if they rely on spurious correlations, i.e., features that are predictive of the target label in the training data but are not causally linked and thus fail to generalize. For the vision domain, many such spurious correlations manifest themselves within the background of the image, where only the foreground is predictive of the class label. In this paper, we introduce Automated Background Swapping (AutoBackSwap) to reduce the reliance of classifiers on such spurious backgrounds. AutoBackSwap uses a secondary network to disentangle the foreground and background, followed by infilling to synthesize complete backgrounds, and finally combines different foregrounds and inpainted backgrounds to augment the training data. We find that patch-wise labeling of just a few hundred samples suffices to train the secondary network and automatically augment the full training dataset on challenging image classification tasks. In contrast to many previous methods, AutoBackSwap proves very effective even if there is not a single sample in the training data breaking the spurious correlation. Across a range of image classification tasks with spurious backgrounds, AutoBackSwap consistently outperforms prior methods.