CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated?
2026-08-17 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors created CaliBench, a new way to test how well video models predict random physical outcomes in a clear, measurable way using known games like dice and roulette. Instead of comparing complicated features, they check if the models produce the correct types of results and how well those results match true probabilities. They found many models often predict just a few outcomes rather than accurately capturing all possibilities, sometimes even collapsing to one. Their work helps better understand and measure uncertainty in video predictions across different scenes and models.
video world modelsaleatoric uncertaintycalibrationtotal variation distancechi-squared testgenerative samplingGalton boardBernoulli forkroulettediscrete outcome space
Authors
Jonathan Sadeghi, Jenny Seidenschwarz, Jesse Allardice, Sirish Srinivasan, Benjamin Graham, Jeffrey Hawke
Abstract
Video world models approximate the stochastic distribution of physical outcomes through generative sampling, but existing benchmarks score individual generations or compare distributions coarsely over a whole dataset, leaving the fine-grained aleatoric uncertainty of specific phenomena untested. We introduce CaliBench, which scores outcomes in a physically interpretable discrete space - a bin index, a die face, a suit, a colour - rather than a learned feature space such as in FID, so the distance from a known reference distribution is measured directly. We curate outcome spaces whose reference is known in closed form (binomial Galton boards, Bernoulli forks, uniform dice/cards/lottery, a skewed European-roulette colour), enabling an exact calibration test. We decompose performance into two orthogonal axes that a single accuracy metric conflates: scorability, the fraction of generations yielding a scoreable outcome, and calibration, the total variation distance from the reference on that sample. A chi-squared test assesses significance; as calibration is its null hypothesis it can evidence only miscalibration, and at N=32 per cell detects only large deviations. We apply it to nine scenes and six image-to-video models (WAN-2.7, SeeDance-2.0, HappyHorse-1.0, Veo 3.1, Runway Gen-4.5, Cosmos3-Super), 32 generations each. Models consistently concentrate probability mass on a few outcomes rather than reproducing the reference. Most scene-model combinations are significantly miscalibrated, in the extreme collapsing to one outcome, as Veo 3.1 does on dice. On roulette, generations often leave the ball ambiguously placed, giving several models low scorability. Performance varies by scene: no model dominates all nine. We release the protocol and a metric (mean normalised total variation, mnTV) for comparing new models against our results.