Distill to Detect: Exposing Stealth Biases in LLMs through Cartridge Distillation
2026-07-01 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors study how language models used in important situations can secretly favor certain ideas or brands, which users can't easily detect. They show that hidden biases can be transferred in subtle ways, making them invisible through normal checks. To solve this, the authors created a method called Distill to Detect (D2D) that amplifies these hidden biases by comparing a suspect model to its original version and turning the differences into clearer signals in the model's output. They also provide a theoretical explanation and prove D2D works well across different bias types, helping auditors find secret preferences in language models.
language modelsbias detectioncontext distillationlogit distributionprefix-tuningKV-cachesoftmax logitsmodel auditingdistributional shift
Authors
Shayan Talaei, Abhinav Chinta, Devvrit Khatri, Amin Karbasi, Azalia Mirhoseini, Amin Saberi
Abstract
Language models deployed in high-stakes roles can potentially favor certain entities, brands, or viewpoints, steering user decisions at scale. Such preferential biases can be introduced by any actor in the model's supply chain and are most dangerous when the model reveals its preference only on the relevant topic while behaving identically to its unmodified base on all other inputs. Recent work has shown that these biases can transfer through context distillation on semantically unrelated data, with the signal residing entirely in the soft logit distribution and remaining invisible to text-based inspection. However, the defender faces a fundamental asymmetry: without knowing the bias topic, no detection method can reliably surface a stealth preferential bias, regardless of whether it examines generated text, internal representations, or model weights. Here we introduce Distill to Detect (D2D), a method that surfaces hidden biases by distilling the distributional shift between a suspected model and its base into a cartridge (a KV-cache prefix adapter), concentrating the dominant divergence and amplifying the bias signal into generated text. We show that D2D successfully amplifies the hidden biases of stealth models to the extent that they can be reliably detected across multiple bias types. We also propose a theoretical framework that explains the efficacy of D2D through the lens of Fisher-weighted projection of the logit distribution shift, supported by empirical observations. By turning the capacity bottleneck of prefix-tuning adapters into a detection tool, D2D provides a practical building block for auditing hidden behaviors in deployed language models.