Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actions
2026-04-08 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors address the challenge of training Android agents using online reinforcement learning, which is slow and inefficient because current methods update actions one at a time from costly emulator states. They propose Android Coach, a new method that lets the agent try multiple actions from the same state to learn better without extra emulator time. This is done by teaching a critic to estimate how good each action is and using that to guide learning. Their experiments show this approach improves success rates and training speed compared to existing methods.
Online reinforcement learningAndroid agentsEmulator latencySingle State Single ActionPolicy updateCritic networkReward modelAdvantage estimatorPPO (Proximal Policy Optimization)Training efficiency
Authors
Guo Gan, Yuxuan Ding, Cong Chen, Yuwei Ren, Yin Huang, Hong Zhou
Abstract
Online reinforcement learning (RL) serves as an effective method for enhancing the capabilities of Android agents. However, guiding agents to learn through online interaction is prohibitively expensive due to the high latency of emulators and the sample inefficiency of existing RL algorithms. We identify a fundamental limitation in current approaches: the Single State Single Action paradigm, which updates the policy with one-to-one state-action pairs from online one-way rollouts without fully exploring each costly emulator state. In this paper, we propose Android Coach, a novel framework that shifts the training paradigm to Single State Multiple Actions, allowing the agent to sample and utilize multiple actions for a single online state. We enable this without additional emulator overhead by learning a critic that estimates action values. To ensure the critic serves as a reliable coach, we integrate a process reward model and introduce a group-wise advantage estimator based on the averaged critic outputs. Extensive experiments demonstrate the effectiveness and efficiency of Android Coach: it achieves 7.5% and 8.3% success rate improvements on AndroidLab and AndroidWorld over UI-TARS-1.5-7B, and attains 1.4x higher training efficiency than Single State Single Action methods PPO and GRPO at matched success rates.