Robust, Scalable Detection of Text Containment in Large Web-Crawled Corpora
2026-07-10 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors created FindMyText, a free tool in Python that checks if a piece of text is inside a big collection of texts. It improves on older methods by looking for sequences of matching parts, which helps it find nearly exact copies instead of just similar texts. This makes it useful for spotting copyrighted content in large datasets like web pages or articles. They tested it on different collections and found it works better than other tools.
document fingerprintingtext corpustext containmentnear-verbatim copy detectionindexed searchdistributed indexingbenchmarkingPython packageweb-crawled datacopyright verification
Authors
Lars Henry Berge Olsen, Pierre Lison, Martin Jullum, Mark Anderson
Abstract
We present FindMyText, an open-source Python package designed to efficiently assess whether a given text appears, in part or in full, within a text corpus. The tool builds on prior techniques for document fingerprinting, but extends them with a novel mechanism to explicitly capture sequences of matching fingerprints. By identifying such chains, the tool can more reliably detect near-verbatim copies of a given text rather than mere textual similarities. This makes FindMyText particularly suited for verifying the presence of copyrighted material in a corpus. Leveraging a distributed, disk-based indexing framework, the system scales to large web-crawled datasets. Using a new benchmark for evaluating text containment methods, we show that FindMyText outperforms alternative approaches across three datasets (ArXiv papers, Wikipedia, and generic web content).