Know Your Source: A Public Knowledge Store for Media Background Checks
2026-07-02 • Computation and Language
Computation and Language
AI summaryⓘ
The authors look at how large language models (LLMs) use external information to check facts, but they point out that this information can sometimes be wrong or biased. To help decide if the sources of this information are trustworthy, the authors focus on something called media background checks (MBCs). Because MBCs usually rely on expensive and private search tools, the authors made a public database called MEDIAREF to make testing easier and cheaper. They show that using MEDIAREF allows LLMs to create better media background checks based on their evaluations.
large language modelsretrieval-augmented generationautomated fact-checkingmedia background checkssource credibilityknowledge storeevidence reliabilityproprietary search APIsevaluation metricsreproducibility
Authors
Benjamin Nichols, Michael Schlichtkrull, Nedjma Ousidhoum
Abstract
LLM-based retrieval-augmented generation (RAG) is increasingly used for automated fact-checking (AFC) and related tasks. By grounding LLM outputs in retrieved evidence, RAG-based systems provide transparent justifications while allowing external information to be updated independently of the underlying model. However, existing approaches often assume retrieved evidence is reliable, although real-world information may be conflicting, outdated, and can originate from unreliable or biased sources. Recent work on *source-critical reasoning* addresses this challenge through media background checks (MBCs) (Schlichtkrull, 2024), which assess the credibility of evidence sources to support downstream fact verification. However, generating MBCs relies on costly proprietary search APIs, limiting reproducibility. To mitigate this issue, we introduce MEDIAREF, a publicly available knowledge store of web-sourced documents that enables reproducible, low-cost evaluation of MBC generation across 200 media sources. We describe a reproducible methodology for constructing and updating the collection, assess widely used LLMs on the MBC generation task, and demonstrate that MEDIAREF supports higher-quality MBC generation through both automatic and qualitative evaluation.