Multi-Camera AR Guidance System for Surgical Instrument Handling and Assembly: Investigating Workload and Efficiency
2026-06-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionHuman-Computer Interaction
AI summaryⓘ
The authors created a system to help scrub nurses during surgery by showing instructions directly in augmented reality (AR) glasses, guiding them on how to handle and assemble surgical instruments. Their system uses multiple cameras to track instrument positions without needing special markers and was trained using synthetic data to work well in real situations. In tests with 29 scrub nurses, the AR guidance made tasks faster and easier, especially for those less familiar with the tools, while keeping error rates similar to using paper manuals. Nurses also felt less overwhelmed and more confident using the AR system.
6D pose estimationaugmented realityscrub nursessurgical instrumentationmulti-camera trackingsynthetic data traininghead-mounted displayknee arthroplastyuser studycognitive workload
Authors
Shiyu Li, Julian Kreimeier, Hannah Schieber, Dirk Müller, Bernhard Kainz, Rüdiger von Eisenhart-Rothe, Daniel Roth
Abstract
The handling and assembly of instruments during surgery imposes high cognitive demands on scrub nurses, particularly when instruments are unfamiliar. We present a supporting guidance system for surgical instrumentation that combines multi-camera 6D pose estimation with augmented reality in-situ visualization on a head-mounted display without the requirement for additional markers. Pose estimation and consecutive camera calibration are achieved through known objects. The 6D pose estimation network is trained purely on synthetic data, aiming for better generalizability and real-world applicability. The AR guidance displays tooltip localization cues and step-wise assembly animations. Via gaze-based selection and a foot pedal, users can switch between assembly steps in intraoperative use. In a technical evaluation, our approach outperforms state-of-art 6D pose estimation. A user study with 29 scrub nurses was conducted in a surgical simulation of knee arthroplasty, comparing the system against a paper manual. AR guidance significantly reduced the perceived workload compared. Objectively, AR guidance reduced task completion time by 21.3\% (4.76 minutes). Specifically, scrub nurses less experienced with the instrument set benefited when using the system. Error frequencies were comparable between conditions. Qualitative feedback highlighted improved process clarity, reduced information overload, and perceived independence. To summarize, our marker-free multi-camera AR guidance approach for surgical instruments can, subjectively and objectively, improve intraoperative instrumentation performance, particularly for untrained scrub nurses.