From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments
2026-06-02 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors propose a new way to understand deep reinforcement learning in settings where actions and states change continuously over time. They model the learning process as a continuous-time random system and focus on an actor-critic algorithm that uses simple neural networks. Their work shows how the environment's state and the learning updates evolve together over different time scales when the network is very wide. They also derive an equation that explains how the state distribution changes during training with very small learning rates. Finally, they test their theory on a simple continuous control problem to show it works.
deep reinforcement learningcontinuous-time stochastic processactor-critic algorithmneural networkstwo time scale processstochastic differential equationsinfinite width limitcontinuous controllearning ratenonparametric formulation
Authors
Saket Tiwari, Tejas Kotwal, George Konidaris
Abstract
We present a novel theoretical framework for deep reinforcement learning (RL) in continuous environments by modeling the problem as a continuous-time stochastic process, drawing on insights from stochastic control. Building on previous work, we introduce a viable model of actor-critic algorithm that incorporates both exploration and stochastic transitions. For single-hidden-layer neural networks, we show that the state of the environment can be formulated as a two time scale process: the environment time and the gradient time. Within this formulation, we characterize how the time-dependent random variables that represent the environment's state and estimate of the cumulative discounted return evolve over gradient steps in the infinite width limit of two-layer networks. Using the theory of stochastic differential equations, we derive, for the first time in continuous RL, an equation describing the infinitesimal change in the state distribution at each gradient step, under a vanishingly small learning rate. Overall, our work provides a novel nonparametric formulation for studying overparametrized neural actor-critic algorithms. We empirically corroborate our theoretical result using a toy continuous control task.