Foundations and Trends® in Machine Learning

1 total work

Risk-Sensitive Reinforcement Learning via Policy Gradient Search

by Prashanth L. A. and Michael C Fu

Published 15 June 2022

Reinforcement learning (RL) is one of the foundational pillars of artificial intelligence and machine learning. An important consideration in any optimization or control problem is the notion of risk, but its incorporation into RL has been a fairly recent development. This monograph surveys research on risk-sensitive RL that uses policy gradient search. The authors survey some of the recent work in this area specifically where policy gradient search is the solution approach. In the first risk-sensitive RL setting, they cover popular risk measures based on variance, conditional value at-risk and chance constraints, and present a template for policy gradient-based risk-sensitive RL algorithms using a Lagrangian formulation. For the setting where risk is incorporated directly into the objective function, they consider an exponential utility formulation, cumulative prospect theory, and coherent risk measures. Written for novices and experts alike the authors have made the text completely self-contained but also organized in a manner that allows expert readers to skip background chapters. This is a complete guide for students and researchers working on this aspect of machine learning.