Marionette: Predicting World States, Rendering Geometry, Painting Appearance
2026-08-14 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors developed Marionette, a model for video game worlds that explicitly tracks and controls the 3D positions and movements of characters, instead of just predicting pixels. They separate the geometric calculations from appearance generation, using a fixed renderer for geometry and a neural network for photorealistic visuals. Their approach lets them better control character actions and fix errors over long times, like reducing unnatural drifting or ground penetration. The visual quality remains high even with this structured approach.
world modelautoregressive dynamics3D articulated skeletonpose controlzero-parameter renderervideo diffusion modelocclusionlong-horizon consistencyphotorealistic synthesisgame character animation
Authors
Zian Meng, Zhen Li, Chuanhao Li, Qiang Li, Kaipeng Zhang
Abstract
Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile. We explicitly model the evolving world state, delegate exact geometric computation to a fixed, zero-parameter renderer, and leave the neural model to synthesize appearance. We instantiate this idea as Marionette, a world model for interactive games with articulated characters. First, a two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. Second, a zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form. Third, a control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls. Our experiments establish two properties of Marionette. First, the predicted world state is directly controllable. Forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments. Second, long-horizon behaviour is determined in the state, and can be repaired there. Left free, the two generated characters drift to 21.2 m apart (recorded sessions stay near 5 m) and a third of frames show ground penetration. Two rules imposed on the explicit state, a terrain collider and a separation cap, cut penetration by 66% and keep the pair engaged, with no change to the observation model. Routing appearance through the predicted state costs no fidelity we can detect, at an FVD of 831 against 799 for recorded pose.