Brand-as-Memory: Vision-Language Models Encode Causal, Mechanistically Localizable Credibility Priors for News Sources
2026-07-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors studied how vision-language models (VLMs) judge the credibility of news articles based on visible source cues like publisher logos and mastheads. They found that these models often rely more on the publisher's identity than the article content itself, a behavior they measured with a new method called CueTrust. This bias varies by model and is linked to specific layers inside the models. The authors showed that the credibility signals come from recognizable visual features and can override actual article information, but this effect can be reduced by targeted interventions. Their work helps understand when and how VLMs might trust source appearance over facts.
vision-language modelssource credibilityCueTrustmastheadlogosource-override indexsignal-magnitude effectmachine learning interpretabilitycontent vs. source bias
Authors
Chih-Ting Liao, Xin Cao
Abstract
Vision-language models (VLMs) increasingly read news and web content as images, where the publisher's identity is visually present. We show that VLMs carry a strong source-credibility prior keyed on outlet identity, and study it along three axes. (i) Cross-model benchmark. We introduce CueTrust, a cross-model diagnostic that measures which surface source cue overrides an article's content evidence via a Source-Override Index (SOI). Across seven VLMs and five cues, the vulnerability profile is model- and scale-dependent, and the override is outlet-identity-specific and encoding-invariant, firing from the masthead name, the logo image, or the bare domain, but not from a named author, in-text authority, or page layout (clean negative controls). (ii) Mechanistic account. For the brand cue, we give a full mechanistic account: swapping only the masthead moves credibility across an approximately 11 log-odds range that tracks professional ratings (rho = 0.88 with Media Bias/Fact Check). The prior is dual-coded (name and logo), strengthens with scale, is causally formed at layers 19-21, carried by interpretable seed-stable sparse-autoencoder features, and recurs at the same relative locus in a second model family. It overrides content (about 1.8x) as a signal-magnitude effect within a shared pathway, not a privileged route. Steering the localized direction selectively reduces the override (41% reduction) and generalizes to held-out outlets, confirming the prior is causally used, not merely decodable. Deployed VLMs may thus defer to source identity over the evidence in front of them, a reliability failure we can measure across models, localize, and causally probe. We release the stimulus suite and CueTrust.