DnA: Denoising Attention for Visual Tasks
2026-06-25 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors point out that the usual softmax method in multihead attention can create confusing focus patterns on images, making the model less accurate. They propose a new method called Denoising Attention (DnA), which uses two types of queries—one to find the correct image parts and another to spot confusing but irrelevant parts. DnA separates these into distinct groups to make the model better at telling important features apart. Their approach improved accuracy on ImageNet and other visual tasks, like video understanding, showing their method helps models focus more clearly.
softmax activationmultihead attentionvisual transformers (ViT)ImageNet-1Kattention patternssubspace separationprincipal anglesvideo transformersvideo large language models (video LLMs)
Authors
Ron Campos, Subhajit Maity, Xin Li, Srijan Das, Aritra Dutta
Abstract
The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy attention patterns that dilute relevant features and degrade its performance. In this paper, we propose Denoising Attention or DnA, in which, first, a positive query identifies which image features belong to the correct class, and a negative query identifies closely associated but irrelevant image features. DnA then projects these interactions into two distinct subspaces with larger principal angles, promoting subspace separation and improved discriminability. Using a ViT-B backbone, our proposed DnA achieves an absolute gain of 0.8% on ImageNet-1K compared to the baseline. We further show improvements across multiple visual understanding tasks, including video understanding with video transformers (1.8%) and video LLMs (0.5%). Our extensive empirical analyses justify the design choices involving two interacting subspaces and the denoising effect of DnA.