Depth-Attention: Cross-Layer Value Mixing for Language Models
2026-06-03 • Computation and Language
Computation and Language
AI summaryⓘ
The authors present Depth-Attention, a new method that improves how Transformers reuse information from earlier layers by letting each layer's attention focus on previous layers' keys and values at the same token position. Unlike earlier methods that add extra hidden states and increase memory use, Depth-Attention fits within the standard attention mechanism without extra parameters or increased cache size. Testing on models with 1.5B and 3B parameters showed better accuracy and lower perplexity compared to traditional Transformers and other cross-layer methods, while using almost the same computational resources. Their improvement works across different model sizes and for looped Transformers.
TransformerSelf-attentionResidual connectionKey-value cacheCross-layer methodsPerplexityDownstream accuracyGrouped-query attentionMulti-head attentionLooped Transformers
Authors
Boyi Zeng, Yiqin Hao, Zitong Wang, Shixiang Song, He Li, Feichen Song, Yifan Liu, Ziwei He, Xinbing Wang, Zhouhan Lin
Abstract
Self-attention selects information freely across the sequence, but across depth, Transformers merely add each layer's output to the residual stream, so later layers cannot selectively reuse earlier-layer representations. Recent cross-layer methods improve this flow but operate on hidden states outside attention, adding state beyond the key-value cache at inference--a cost that becomes increasingly salient as modern LLMs compress the cache with grouped-query and multi-head latent attention. We introduce Depth-Attention, which performs this selection inside the attention module itself: before a layer attends over the sequence, its query attends over the keys of earlier layers at the same token position and mixes their values into the value that self-attention then reads. Because Depth-Attention reuses the standard attention queries, keys, and value-cache slots, storing depth-mixed values in place of the original values, it adds no parameters and introduces no persistent inference state beyond the standard key-value cache--the same cache size as a vanilla decoder and less than hidden-state-based cross-layer methods. On Qwen3-style decoders at 1.5B and 3B parameters, Depth-Attention attains the lowest perplexity and the highest average downstream accuracy, improving over the vanilla Transformer by up to 2.3 accuracy points and surpassing strong cross-layer baselines in perplexity and average accuracy, while adding under 0.01% extra arithmetic FLOPs and no additional persistent inference state. The gains hold from 360M to 3B parameters and extend to looped Transformers.