Prop-Chromeleon: Adaptive Haptic Props in Mixed Reality through Generative Artificial Intelligence
2026-05-01 • Human-Computer Interaction
Human-Computer Interaction
AI summaryⓘ
The authors created a Mixed Reality system called Prop-Chromeleon that uses AI to turn real objects into virtual ones that feel more real by matching their shape and appearance based on user descriptions. Their system generates digital content that fits the physical objects and tested how well it works with different shapes and prompts. They found that their method makes the experience feel more realistic, immersive, and fun compared to fixed virtual objects. This shows that using AI to shape virtual items based on real objects can improve how people interact with and enjoy mixed reality.
Mixed RealityHaptic FeedbackGenerative AIVirtual AssetsPassive Haptic PropsShape Similarity MetricsPrompt FidelityUser StudyImmersion
Authors
Haoyu Wang, Fengyuan Zhu, Bingjian Huang, Zhecheng Wang, Ludwig Sidenmark
Abstract
Mixed Reality (MR) aims to blend digital and physical worlds, but the absence of haptic feedback often breaks visual-tactile consistency. We introduce Prop-Chromeleon, a MR system based on generative artificial intelligence (AI) that dynamically transforms everyday objects into adaptive passive haptic props through user-provided text prompts. Our AI pipeline performs generation and anchoring of virtual assets that align with the shape of physical props, allowing us to study how virtual content generation behaves under geometric and prompt-based constraints. We evaluate Prop-Chromeleon's effectiveness through a generation study using varied object shapes and user prompts, combining quantitative shape similarity metrics with qualitative prompt fidelity analysis. Our user study further showcases Prop-Chromeleon's improvements in perceived realism, immersion, and enjoyment compared to static baselines. These results show that shape-aware generation can support both believable haptic interaction and creative engagement in MR.