CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators
2026-08-27 • Robotics
RoboticsArtificial IntelligenceComputer Vision and Pattern Recognition
AI summaryⓘ
The authors developed CLAP, a system that can learn to generate action-based videos involving different types of robots and humans by understanding shared physical rules. Since robot actions vary widely and human action labels are often missing, their approach uses a mix of robot hand movements, language instructions, and hidden action codes to bridge these differences. They train CLAP in stages, first learning general physical patterns from unlabeled videos and then applying this knowledge to specific robots without extra training. Their model performs as well or better than current models that only focus on one robot type and works across various robot bodies and action inputs.
action-conditioned video modelscross-embodiment learningend-effector poselatent actionscurriculum learningzero-shot deploymentphysical priorsrobot morphologiesfew-shot adaptationvideo world models
Authors
Kechen Liu, Ola Shorinwa
Abstract
State-of-the-art action-conditioned video models are typically restricted to a single robot embodiment, preventing them from leveraging the vast corpus of heterogeneous video data that contains rich signals for learning generalizable physics. To bridge this gap, we introduce CLAP, a framework for cross-embodiment action-conditioned video generation capable of being trained on diverse, internet-scale videos across human and robotic agents. CLAP is grounded in the insight that universal physical laws govern spatiotemporal dynamics regardless of the actor. However, cross-embodiment learning is non-trivial because action representations vary sharply across robot platforms and are typically absent in human videos. CLAP addresses this fundamental challenge through the following core contributions. First, CLAP reconciles disparate action spaces using end-effector poses, language instructions, and latent actions. Second, to resolve their individual limitations, CLAP introduces a curriculum-based cross-embodiment learning recipe that first learns foundational physical priors across unlabeled video data using latent actions and subsequently grounds them in end-effector action spaces for zero-shot deployment to real-world tasks. Crucially, CLAP approaches or surpasses state-of-the-art single-embodiment video models in challenging environments like DROID. These performance advantages compound via few-shot adaptation to establish a novel paradigm for training single-embodiment video world models. Ultimately, CLAP delivers the most comprehensive suite of action-conditioned video world models to date - spanning diverse action-conditioning spaces (end-effector, language, and latent) and robot morphologies (including cross-embodiment, DROID, Bridge, bimanual YAM robots, and G1 humanoids). We open-source all code and models. Project Website at https://omni-clap.github.io .