Issue |
ESAIM: COCV
Volume 29, 2023
|
|
---|---|---|
Article Number | 20 | |
Number of page(s) | 26 | |
DOI | https://doi.org/10.1051/cocv/2023007 | |
Published online | 07 March 2023 |
Variational actor-critic algorithms*,,**
1
Department of Mathematics and Halicioğlu Data Science Institute, University of California,
San Diego, La Jolla,
California, USA
2
Department of Mathematics, Stanford University,
Stanford,
California, USA
*** Corresponding author: yuhuazhu@stanford.edu
Received:
2
August
2021
Accepted:
13
January
2023
We introduce a class of variational actor-critic algorithms based on a variational formulation over both the value function and the policy. The objective function of the variational formulation consists of two parts: one for maximizing the value function and the other for minimizing the Bellman residual. Besides the vanilla gradient descent with both the value function and the policy updates, we propose two variants, the clipping method and the flipping method, in order to speed up the convergence. We also prove that, when the prefactor of the Bellman residual is sufficiently large, the fixed point of the algorithm is close to the optimal policy.
Mathematics Subject Classification: 90C40 / 93E20
Key words: Markov decision process / reinforcement learning / policy gradient / optimal control
© The authors. Published by EDP Sciences, SMAI 2023
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Current usage metrics show cumulative count of Article Views (full-text article views including HTML views, PDF and ePub downloads, according to the available data) and Abstracts Views on Vision4Press platform.
Data correspond to usage on the plateform after 2015. The current usage metrics is available 48-96 hours after online publication and is updated daily on week days.
Initial download of the metrics may take a while.