Risk-Aware General-Utility Markov Decision Processes
2026-07-10 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study a type of decision-making process where an agent cares not just about the average results but also about risk, focusing on how often states are visited. They introduce a formal way to handle risk in these problems using something called the entropic risk measure. To find good strategies, they use a planning method called Monte Carlo Tree Search that can guarantee solutions close to the best possible. Their experiments show this method works well for different tasks, including exploring environments, learning by copying, and handling multiple goals.
Markov decision processrisk-awaregeneral-utility MDPentropic risk measureMonte Carlo Tree Searchonline planningstate visitation frequencyimitation learningmulti-objective optimizationmaximum state entropy
Authors
Pedro P. Santos, Fábio Vital, Alberto Sardinha, Francisco S. Melo
Abstract
We study general-utility Markov decision processes (GUMDPs) with risk-aware objectives. In this framework, an agent aims to optimize a risk measure of the distribution of objective values, where the objective function depends on the frequency of visitation of states induced by the agent's policy. First, we motivate, propose, and formalize risk-aware GUMDPs, which enable agents and decision makers to trade off expected performance by risk aversion while benefiting from the rich set of objectives that can be cast under the framework of GUMDPs. We focus our attention on the entropic risk measure (ERM). Second, we show how we can solve risk-aware GUMDPs with ERM objectives by resorting to online planning techniques. In particular, we propose an approach based on Monte Carlo Tree Search (MCTS) to provably solve risk-aware GUMDPs up to any desired accuracy. Third, we provide a set of experimental results showcasing that our approach is successful when optimizing for a spectrum of risk-aware behaviors in the context of GUMDPs under diverse tasks (standard MDPs, maximum state entropy exploration, imitation learning, and multi-objective MDPs).