Pretraining Data Can Be Poisoned through Computational Propaganda
2026-07-16 • Artificial Intelligence
Artificial IntelligenceComputation and Language
AI summaryⓘ
The authors study how bad actors can sneak harmful content into the huge amounts of text that teach language models (LMs) how to understand and generate language. They point out that past research focused mostly on well-known, stable sources like Wikipedia, but real training data often comes from many kinds of websites. The authors show that public comment sections on websites can be used to insert bad content into training data. They also create a tool called HalfLife to check if this bad content actually ends up in the final training sets after web crawling and cleaning. Their work highlights the risk of using third-party web content when building language models.
Language ModelsData PoisoningPretraining DataWeb CrawlingData CurationContent InjectionPublic Discussion InterfacesAdversarial ContentHalfLife Analysis
Authors
Victoria Graf, Hannaneh Hajishirzi, Noah A. Smith, David Kohlbrenner, Kyle Lo
Abstract
Poisoning pretraining data can introduce harmful behaviors to LMs that are difficult to detect and mitigate. Prior work on poisoning pretraining data has largely exploited established data sources such as Wikipedia, which do not represent the large scale and heterogeneity typical of pretraining corpora, and has ignored the interaction between poisoned data and data curation pipelines. We demonstrate that poisoning attacks on pretraining data are feasible beyond this limited setting through an existing web-scale content injection mechanism: public discussion interfaces. Additionally, to measure whether malicious content is included after web crawling and data curation, we introduce HalfLife, a novel analysis for estimating adversarial content inclusion in web-crawl based LM training data. We use HalfLife to explore the feasibility of poisoning pretraining corpora at web scale through open discussion interfaces. Our analysis demonstrates the importance of estimating whether poison injections are included in pretraining data, and establishes third-party webpage content as a possible vector for attacking language model pretraining.