AI summaryⓘ
The authors studied a new way that bad actors could secretly mess with science by hacking open data used by AI systems, a method they call indirect data poisoning. They showed that when AI research agents pull from corrupted public datasets, nearly half of the experiments were tricked into wrong conclusions, but only a small fraction were caught. This form of attack doesn't rely on obvious clues, making it hard to detect automatically. To stop this, the authors tested methods like role-playing a careful scientist and checking the origins and consistency of data, finding that careful data checking can fully prevent the fraud. Their work highlights a potential risk as AI tools become common in research, and suggests audits during data use are crucial to maintain trust.
Scientific fraudIndirect data poisoningArtificial intelligence (AI)Autonomous research agentsDataset poisoningData provenanceAI auditOpen data ecosystemScientific integrityMetadata manipulation
Abstract
Scientific fraud is the instrument of doubt that malicious entities can use to establish controversy in science. Historically, it required the resources of a company: deep pockets, ghostwritten articles, and corrupt academics. Today, Artificial Intelligence (AI) is increasingly automating scientific research, so we ask: Can a remote adversary weaponize the honest use of AI in science to compromise scientific integrity? We envision and empirically evaluate a new attack, indirect data poisoning, in which an adversary corrupts an open dataset and uploads the poisoned variant to a public repository. Autonomous research agents may independently retrieve and process this data, turning honest scientists into the unpaid and unwitting distributors of fraud at scale. Across five socially-salient topics, from hiring discrimination to the safety of autonomous vehicles, three widely used frontier AI systems (Claude Code with Claude Opus 4.7, Codex with GPT-5.5, Gemini CLI with Gemini 3.1 Pro), and 450 ethically contained experimental runs, we find that poisoning succeeds in 49.56% of runs, while the rate of poisoning detection is only 6.0%. The attack requires no topic-specific trigger-words, agent access, indirect prompt injection, or fabricated papers, only the open data ecosystem and misleading metadata. To mitigate the attacks, we propose and evaluate two measures: a scientist persona and a data provenance audit with five checks (referencing papers, social markers, statistical anomalies, related datasets, poisoning caution). We find that the persona still leaves 16.67% of runs with a poisoned conclusion, but provenance auditing reduces attack success rate to zero. Our results suggest that indirect data poisoning may enable scientific fraud at unprecedented scale, but these attacks can be mitigated with suitable auditing by agents during data retrieval.