SPECTRA: Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics
2026-05-29 • Information Retrieval
Information RetrievalArtificial Intelligence
AI summaryⓘ
The authors created SPECTRA, a tool that makes fake but realistic text collections to test search engines. This helps check how well search systems work when they have lots of documents, without needing expensive real human-marked data. Their tests show SPECTRA can quickly make big collections with controllable features like topic variety and word frequency. The authors see this tool as a helpful way to find problems early in search engine designs, not as a full replacement for real human assessments.
information retrievalsynthetic corporatest collectionstopic modelingquery intentrelevance labelsBM25nDCGZipf's lawevaluation metrics
Authors
Eric Liang
Abstract
Scalable information retrieval testing needs corpora that are large enough to stress index construction, ranking latency, query routing, and evaluation tooling, yet human-judged test collections remain expensive and may be unavailable when documents are private or still under design. This paper introduces SPECTRA, a reproducible framework for generating synthetic text corpora and retrieval test collections through a separation of latent topical structure, surface text realization, metadata controls, query intent generation, and deterministic relevance oracles. The framework is intended as a diagnostic complement to Cranfield-style and TREC-style evaluation, not as a replacement for human assessment. A single-process Python prototype generated corpora up to 60,000 documents and 9.61 million tokens while preserving controllable long-tail vocabulary growth and producing graded relevance labels for 96 queries. In the local simulation study, generation remained close to linear at roughly 12K to 14K documents per second, estimated Zipf slopes stayed near 0.86 in absolute value, and increasing cross-topic distractor text reduced BM25 nDCG@10 from 1.00 at 2% distractors to 0.43 at 36% distractors. These results show that lightweight synthetic corpora can expose retrieval-system scaling and failure modes before costly collection construction begins.