VERITAS: Towards a General-Purpose Replication Tool for Scientific Research
2026-07-03 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created VERITAS, a tool that helps automatically check and repeat the experiments in scientific papers using coding agents. Unlike previous tools, VERITAS works across different fields and can handle papers, code, or both. It tests the claims made in a paper by running the experiments, fixing problems, and then scores how well the results can be replicated. They tested VERITAS on 65 papers from various disciplines and found it performed better than existing methods using similar AI models. This helps make scientific verification faster and more reliable.
replicationcoding agentsbenchmarkscientific publicationAI toolsCLAIM extractioncode repositoryexperiment automationreplication scorenatural language processing
Authors
Haokun Liu, Filbert Aurelian Tjiaranata, Chenhao Tan
Abstract
AI tools are accelerating scientific publication while the systems that review it struggle to keep up, and independent verification of published research has become both harder and more important. As manual replication is slow and expensive, a growing line of work uses coding agents to automate parts of the process. Existing efforts are largely packaged as benchmarks with companion agents that only run inside the benchmark's own pipeline, and no general-purpose replication tool exists. We present VERITAS, a domain-agnostic replication framework built around CLI coding agents. Given a paper, a code repository, or both, VERITAS extracts the paper's claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs. The pipeline returns an importance-weighted Replication Score, a severity-rated log of every fix applied, and the patched codebase. We evaluate VERITAS on CORE-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics. Against two strong Claude Code baselines on the same model and host environment, VERITAS achieves state-of-the-art performance and leads on every metric on both benchmarks.