CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving
2026-07-03 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors focus on improving autonomous driving systems that use raw sensor data to control cars directly. They note that current models combining vision, language, and action mainly learn by copying expert drivers, which struggles when actually driving in new situations. To fix this, they introduce CLEAR, a system that uses reinforcement learning to train the car’s decision-making in real-time, building on existing models by refining planned routes. They also optimize computing resources to run many simulations simultaneously, helping the system learn faster and perform better in tough driving tests.
End-to-end autonomous drivingMulti-modal large language modelsVision-Language-Action modelsImitation learningReinforcement learningClosed-loop trainingWaypoint policySimulation environmentsCARLA benchmark
Authors
Yunxiao Shi, Hong Cai, Mohammad Ghavamzadeh, Fatih Porikli
Abstract
End-to-end autonomous driving (E2E-AD) aims to directly map raw sensor information to driving actions. Recently, with the rapid advancement of multi-modal large language models (MLLMs), researchers have proposed the paradigm of Vision-Language-Action (VLA) models for E2E-AD, where it seeks to integrate visual perception, language understanding and action prediction within a single policy. However, existing VLA-based policies largely adopts imitation learning, where it only learns to drive by optimizing distance-based metrics w.r.t. logged expert trajectories. Such distribution shift between open-loop training and closed-loop inference leads to suboptimal performance in closed-loop planning. To close this gap, we present CLEAR, a system that enables closed-loop training using Reinforcement Learning (RL) at scale for E2E-AD. We propose to learn a novel residual waypoint policy around the waypoint prior from pretrained VLA policies, effectively harnessing the knowledge within. On another front, one of the key challenges to scale up RL for vision-based policies is the number of parallel simulation environments since RL is data hungry. To that end, we design a heterogeneous pipeline that places the simulator and the VLA learner on distinct compute groups, which allows us to dramatically increase the number of simulation environments running in parallel while avoiding resource contention and maintaining training stability. We show that with a simple reward, CLEAR significantly outperforms previous methods and sets new state-of-the-art performance on the challenging benchmarks of CARLA longest6 v2 and Bench2Drive.