Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini
2026-03-25 • Human-Computer Interaction
Human-Computer Interaction
AI summaryⓘ
The authors created XR Blocks, a free and easy-to-use toolkit that simplifies building extended reality (XR) experiences by hiding complicated technical details. They also designed Vibe Coding XR, a system that uses large language models to turn simple text instructions into working XR applications quickly through a web interface. Their tests show this approach can make interactive XR projects in less than a minute, helping people create XR software without needing deep technical skills. They provide examples and code online for others to try.
Extended Reality (XR)WebXRLarge Language Models (LLMs)Rapid PrototypingSpatial ComputingHuman-Centered DesignGenerative AIMulti-modal InteractionOpen-source FrameworkGame Engines
Authors
Ruofei Du, Benjamin Hersh, David Li, Nels Numan, Xun Qian, Yanhe Chen, Zhongyi Zhou, Xingyue Chen, Jiahao Ren, Robert Timothy Bettridge, Steve Toh, David Kim
Abstract
While large language models have accelerated software development through "vibe coding", prototyping intelligent Extended Reality (XR) experiences remains inaccessible due to the friction of complex game engines and low-level sensor integration. To bridge this gap, we contribute XR Blocks, an open-source, modular WebXR framework that abstracts spatial computing complexities into high-level, human-centered primitives. Building upon this foundation, we present Vibe Coding XR, an end-to-end rapid prototyping workflow that leverages LLMs to translate natural language intent directly into functional XR software. Using a web-based interface, creators can transform high-level prompts (e.g., "create a dandelion that reacts to hand") into interactive WebXR applications in under a minute. We provide a preliminary technical evaluation on a pilot dataset (VCXR60) alongside diverse application scenarios highlighting mixed-reality realism, multi-modal interaction, and generative AI integrations. By democratizing spatial software creation, this work empowers practitioners to bypass low-level hurdles and rapidly move from "idea to reality." Code and live demos are available at https://xrblocks.github.io/gem and https://github.com/google/xrblocks.