Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions
2026-08-19 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how to control systems influenced by complex, memory-dependent random events called multivariate Hawkes processes, which are tricky because they depend on their past in complicated ways. They create a method to simplify these processes by approximating them with models that are easier to handle (Markovian), and they prove this simplification is accurate. Using this, they develop a continuous-time reinforcement learning algorithm that can learn to control the system by just observing events and the system's response, without knowing all the details of the underlying process. They test their method against other reinforcement learning approaches with different shapes of memory effects (kernels).
Stochastic controlMultivariate Hawkes processesNon-Markovian dynamicsMarkovian approximationStochastic differential equationsReinforcement learningPolicy gradientContinuous-time learningKernel functionsEvent-driven systems
Authors
Tomasz R. Bielecki, Thibaut Mastrolia, Haoze Yan
Abstract
We study stochastic control of multivariate Hawkes-driven stochastic differential equations with machine learning algorithms in a non-Markovian setting. Due to the path dependence of the memory of the Hawkes intensity, this problem does not fall within classical stochastic control theory outside particular Markovian kernels. We first develop a finite-dimensional Markovianization procedure and algorithm to approximate multivariate Hawkes processes with mixtures of exponential kernels. We prove the convergence of the Markovianized approximation of the Hawkes process, its intensity, and the value of the problem to the original non-Markovian processes and the value of the primal problem. We then formulate continuous-time deterministic policy gradient learning on the Markovianized approximation of the problem, called Hawkes-CT DDPG. We propose a model-free algorithm to solve the non-Markovian Hawkes-driven optimization by observing only the event times of the process, the realization of the solution to the SDE, and a chosen set of decay filters, while the Hawkes kernel coefficients remain unknown. We compare our continuous time reinforcement learning Hawkes-CT DDPG method with discrete time reinforcement learning techniques under three different types of kernels: simple exponential, Erlang, and power-law kernels.