RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature
2026-08-27 • Computation and Language
Computation and LanguageInformation Retrieval
AI summaryⓘ
The authors created RATIO, a new benchmark to help computers find useful scientific papers in different ways. They defined three types of helpful search methods: finding ways to solve a problem (Address), finding broader ideas (Broaden), and finding specific examples (Specify). RATIO was built using millions of computer science papers and tested with both AI and humans to check quality. Their tests showed that tuning search tools for each method works better but still has room to improve. This benchmark can help develop better tools to inspire scientific research by retrieving relevant literature.
information retrievalbenchmarkscientific literaturelarge language modelsfine-tuningdiscourse markerscorpus-scale retrievalnatural language processingideation moves
Authors
Maayan Sharon, Tom Hope
Abstract
Retrieved scientific literature can serve as inspiration for both human and AI scientists. Inspiration can take different forms: prior work may directly suggest how to address a problem, or surface directions at different levels of abstraction - zooming out to a more general view or zooming in to a concrete realization. We introduce RATIO (Retrieval Across Typed Ideation Operations), a large-scale benchmark in which relevance is defined by three operations which we name ideation moves: Address retrieves potential approaches for stated problems, Broaden retrieves more general formulations, and Specify retrieves concrete instantiations. RATIO is constructed from millions of full-text scientific papers across CS literature via a general recipe that extends discourse-marker distant supervision - previously used only for classification - to corpus-scale retrieval, combined with extensive LLM and human vetting. Experiments show that operation-specific fine-tuning substantially boosts retrievers but leaves much room for further improvements. RATIO provides a scalable training and evaluation framework for retrieval components that support literature-grounded ideation, opening up new research avenues on scientific inspiration retrieval.