How Retrieved Context Shapes Internal Representations in RAG
2026-02-23 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how large language models (LLMs) change their internal 'thought process' when they use extra documents to help answer questions, especially when those documents aren’t all equally useful. They looked inside the model’s hidden layers to see how different types of retrieved documents affect what the model 'thinks' before giving an answer. Their research found that how relevant the documents are, and how the model processes them across its layers, impacts both the internal representations and the final answers. This helps explain why LLMs behave the way they do when using external information and offers advice for building better retrieval systems.
Retrieval-augmented generationLarge language modelsInternal representationsHidden statesContext relevancyInformation integrationQuestion answeringLayer-wise processing
Authors
Samuel Yeh, Sharon Li
Abstract
Retrieval-augmented generation (RAG) enhances large language models (LLMs) by conditioning generation on retrieved external documents, but the effect of retrieved context is often non-trivial. In realistic retrieval settings, the retrieved document set often contains a mixture of documents that vary in relevance and usefulness. While prior work has largely examined these phenomena through output behavior, little is known about how retrieved context shapes the internal representations that mediate information integration in RAG. In this work, we study RAG through the lens of latent representations. We systematically analyze how different types of retrieved documents affect the hidden states of LLMs, and how these internal representation shifts relate to downstream generation behavior. Across four question-answering datasets and three LLMs, we analyze internal representations under controlled single- and multi-document settings. Our results reveal how context relevancy and layer-wise processing influence internal representations, providing explanations on LLMs output behaviors and insights for RAG system design.