Individual Parameters in Weight-Sparse Transformers Appear Interpretable
2026-07-03 • Machine Learning
Machine LearningArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors studied how to understand what individual weights in neural networks do across all types of inputs, instead of just focusing on specific behaviors. They created an automated process that writes simple descriptions of when a weight matters by checking if removing it changes the model's output. Their findings show that in sparse transformer models, many weights (12-31%) have clear, general roles that can be described simply, more so than in dense transformers. This suggests some weights in sparse models have consistent functions across different inputs.
mechanistic interpretabilityneural networkstransformerssparse transformersdense transformersweightsmodel ablationinput distributionlarge language models (LLMs)circuit-finding
Authors
Arnau Marin-Llobet, Stefan Heimersheim
Abstract
A central goal of mechanistic interpretability is to understand how neural networks work and what each individual component does. Dominant circuit-finding approaches focus on a specific behavior and reverse-engineer the role of components on the associated sub-distribution. However, past work has shown that components can have different functions that are active on different subsets of the input distribution. In this work we ask whether a single weight can be understood globally across the full training distribution by characterizing when it matters (the inputs on which ablating it changes the model's predictions). We introduce an automated LLM pipeline that writes a short, human-readable description of when a weight matters and verifies it on held-out text, crediting a weight only if its description generalizes. Across two sparse and two dense transformers, the fraction of weights that are interpretable (in this sense) is higher in sparse transformers than in dense ones, a gap that widens once unreliable descriptions are discarded. Our results show that a meaningful fraction of a sparse transformer model's weights can be interpreted: 12 to 31% of weights have a single short description that identifies what the weight is used for.