COP-Q: Safety-First Reinforcement Learning for Robot Control via Cholesky-Ordered Projection
2026-06-03 • Robotics
RoboticsMachine Learning
AI summaryⓘ
The authors address how robots can learn to perform tasks safely while still trying to do well. They notice that existing methods treat safety and reward separately, which can make the learning too cautious and slow. Their new method, COP-Q, considers both safety and reward together, allowing the robot to be careful but not overly cautious. Tests show their approach helps robots learn safely and efficiently in different environments.
safe reinforcement learningQ-learningoff-policy learningcritic ensemblesconfidence boundsCholesky factorizationtemporal-difference learningactor optimizationsample efficiencyrobot locomotion
Authors
Guopeng Li, Moritz A. Zanger, Matthijs T. J. Spaan, Julian F. P. Kooij
Abstract
Safe robot control requires maximizing return while satisfying safety constraints. In off-policy safe reinforcement learning, reward and safety Q-values are commonly learned by separate critic ensembles, with uncertainty handled independently for each objective. This objective-wise treatment neglects inter-objective correlation and can lead to overly conservative value estimates, thereby reducing sample efficiency. To address this issue, we propose Cholesky-Ordered Projection Q-learning (COP-Q), a safety-first method that incorporates inter-objective covariance into vector-valued Q-value estimation. COP-Q constructs a generalized confidence bound in the joint Q-value space and uses Cholesky factorization to encode objective priority in a sequential form. This preserves conservatism on safety while adaptively reducing excessive conservatism on the reward objective. The resulting estimate is used in both temporal-difference target computation and actor optimization. COP-Q incurs minimal computational overhead and is readily compatible with most existing deep Q-learning frameworks. Experiments on robot locomotion in Brax and safe navigation in Safety-Gymnasium, covering both hard- and soft-safety settings, demonstrate that COP-Q achieves strong safety performance together with competitive or improved sample efficiency relative to representative baselines.