World Mouse: Exploring Interactions with a Cross-Reality Cursor
2026-03-11 • Human-Computer Interaction
Human-Computer Interaction
AI summaryⓘ
The authors present World Mouse, a new way to use a 2D mouse cursor for interacting with both real and virtual 3D environments. They solve two main problems: how to precisely place the cursor on objects and how to move the cursor through empty space between objects. Unlike previous methods that only worked in virtual worlds, their system uses advanced techniques to recognize and interact with physical objects too. Their prototypes show that this approach helps users smoothly work across real and virtual spaces.
Extended Reality2D Mouse Cursor3D InteractionSurface NormalsSemantic SegmentationMesh ReconstructionCross-RealityCursor NavigationHuman-Computer Interaction
Authors
Esen K. Tütüncü, Mar Gonzalez-Franco, Khushman Patel, Eric J. Gonzalez
Abstract
As Extended Reality (XR) systems increasingly map and understand the physical world, interacting with these blended representations remains challenging. The current push for "natural" inputs has its trade-offs: touch is limited by human reach and fatigue, while gaze often lacks the precision for fine interaction. To bridge this gap, we introduce World Mouse, a cross-reality cursor that reinterprets the familiar 2D desktop mouse for complex 3D scenes. The system is driven by two core mechanisms: within-object interaction, which uses surface normals for precise cursor placement, and between-object navigation, which leverages interpolation to traverse empty space. Unlike previous virtual-only approaches, World Mouse leverages semantic segmentation and mesh reconstruction to treat physical objects as interactive surfaces. Through a series of prototypes, including object manipulation and screen-to-world transitions, we illustrate how cross-reality cursors may enable seamless interactions across real and virtual environments.