Pictura: Perspective-View Self-Play at Scale for Driving
2026-07-28 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceRobotics
AI summaryⓘ
The authors created Pictura, a fast driving simulator that shows what each car 'sees' through its own cameras instead of using perfect hidden information. This helps train self-driving policies in a way that matches what real cars would experience, avoiding the problem of teaching from unrealistic views. They trained a driving policy called Alberti using only these realistic camera views and found it performed nearly as well as policies trained with perfect information. Alberti also worked well when tested on new driving scenarios from a well-known dataset without extra adjustment.
self-playegocentric cameradriving policymulti-agent simulationPPO (Proximal Policy Optimization)vectorized observationsrepresentation gapzero-shot transferGPU accelerationWaymo Open Motion Dataset
Authors
Yuan Yin, Elias Ramzi, Marc Lafon, Valentin Charraut, Victor Bares, Yihong Xu, Éloi Zablocki, Alexandre Boulch, Thibault Buhet, Andrei Bursuc, Matthieu Cord
Abstract
Self-play in simulation produces robust driving policies at scale. Demonstrations of such behavior have been made using privileged vectorized observations such as exact poses and velocities, even for occluded agents. This assumes that perception is solved and introduces a representation gap with the partial observation of a deployed agent driving from the perspective view of egocentric cameras. A common fix, distilling the privileged policy into a camera-input student, leaves the student imitating decisions its own view cannot justify. Instead, we establish perspective-view self-play as a practical training regime. We introduce Pictura, a GPU-accelerated multi-agent driving simulator that renders each agent's egocentric view at every step, mitigating the representation gap at its source. Pictura sustains up to 500K agent-steps/s (2M images/s) on a single H100. Using Pictura, we train Alberti by self-play with plain PPO. It is the first large-scale driving self-play policy trained directly from perspective images, without privileged observations. Training spans 50B agent steps for ~35M km of driving. It approaches the driving performance of its privileged vectorized counterpart, and transfers zero-shot to Waymo Open Motion Dataset layouts re-rendered in Pictura, where it outperforms privileged vectorized agents. Project page: https://valeoai.github.io/Pictura/