Can Current Agents Close the Discovery-to-Application Gap? A Case Study in Minecraft
2026-04-27 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created SciCrafter, a Minecraft-based test that challenges AI to discover and use new knowledge by building complex redstone circuits that light lamps in patterns. They tested advanced AI models but found all struggled to solve more than a quarter of the tasks. By breaking down the problem into four parts, they saw that while applying knowledge is still the hardest step, the ability to identify what needs to be learned is becoming a bigger barrier. The authors offer SciCrafter as a tool for studying how AI can better handle the full process from discovery to practical use.
causal discoveryredstone circuitsMinecraft benchmarkgeneral intelligenceknowledge applicationexperimental discoveryknowledge consolidationAI evaluationproblem identificationcode agents
Authors
Zhou Ziheng, Huacong Tang, Jinyuan Zhang, Haowei Lin, Bangcheng Yang, Qian Long, Fang Sun, Yizhou Sun, Yitao Liang, Ying Nian Wu, Demetri Terzopoulos, Xiaofeng Gao
Abstract
Discovering causal regularities and applying them to build functional systems--the discovery-to-application loop--is a hallmark of general intelligence, yet evaluating this capacity has been hindered by the vast complexity gap between scientific discovery and real-world engineering. We introduce SciCrafter, a Minecraft-based benchmark that operationalizes this loop through parameterized redstone circuit tasks. Agents must ignite lamps in specified patterns (e.g., simultaneously or in timed sequences); scaling target parameters substantially increases construction complexity and required knowledge, forcing genuine discovery rather than reliance on memorized solutions. Evaluating frontier models including GPT-5.2, Gemini-3-Pro, and Claude-Opus-4.5 under a general-purpose code agent scaffold, we find that all plateau at approximately 26% success rate. To diagnose these failures, we decompose the loop into four capacities--knowledge gap identification, experimental discovery, knowledge consolidation, and knowledge application--and design targeted interventions whose marginal contributions serve as proxies for corresponding gaps. Our analysis reveals that although the general knowledge application capability still remains as the biggest gap across all models, for frontier models the knowledge gap identification starts to become a major hurdle--indicating the bottleneck is shifting from solving problems right to raising the right problems for current AI. We release SciCrafter as a diagnostic probe for future research on AI systems that navigate the full discovery-to-application loop.