Proximal policy optimization with adaptive generalized advantage estimate: critic-aware refinements

Document Type : Research Article

Authors

1 Department of Mechanical Engineering, Amirkabir University of Technology, Tehran, Iran

2 Department of Computer Engineering, Amirkabir University of Technology, Tehran, Iran

Abstract

Proximal Policy Optimization (PPO) is one of the most widely used methods in reinforcement learning, designed to optimize policy updates while maintaining training stability. However, in complex and high-dimensional environments, maintaining a suitable balance between bias and variance poses a significant challenge. The λ parameter in Generalized Advantage Estimation (GAE) influences this balance by controlling the trade-off between short-term and long-term return estimations. In this study, we propose a method for adaptive adjustment of the λ parameter, where λ is dynamically updated during training instead of remaining fixed. The updates are guided by internal learning signals such as the value function loss and Explained Variance—a statistical measure that reflects how accurately the critic estimates target returns. To further enhance training robustness, we incorporate a Policy Update Delay (PUD) mechanism to mitigate instability from overly frequent policy updates. The main objective of this approach is to reduce dependence on expensive and time-consuming hyperparameter tuning. By leveraging internal indicators from the learning process, the proposed method contributes to the development of more adaptive, stable, and generalizable reinforcement learning algorithms. To assess the effectiveness of the approach, experiments are conducted in four diverse and standard benchmark environments: Ant-v4, HalfCheetah-v4, and Humanoid-v4 from the OpenAI Gym, as well as Quadruped-Walk from the DeepMind Control Suite. The results demonstrate that the proposed method can substantially improve the performance and stability of PPO across these environments. Our implementation is publicly available at https://github.com/naempr/PPO-with-adaptive-GAE.

Keywords

References

[1] S. Fujimoto, H. Hoof, D. Meger, Addressing function approximation error in actor-critic methods, In International conference on machine learning (2018) 1587–1596.
[2] N. Heess, G. Wayne, D. Silver, T. Lillicrap, T. Erez, Y. Tassa, Learning continuous control policies by stochastic value gradients, Adv. Neural Inf. Process. Syst. 28 (2015) 2944–2952.
[3] S. Kakade, J. Langford, Approximately optimal approximate reinforcement learning, In Proceedings of the nineteenth international conference on machine learning (2002) 267–274.
[4] T.P. Lillicrap, J.J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra, Continuous control with deep reinforcement learning (2015) arXiv:1509.02971.
[5] V. Mnih, K. Kavukcuoglu, D. Silver, A.A. Rusu, J. Veness, M.G. Bellemare, A. Graves, M. Riedmiller, A.K. Fidjeland, G. Ostrovski, Human-level control through deep reinforcement learning, Nature 518 (2015) 529–533.
[6] N. Mohammadpour, M. Fozi, M.M. Ebadzadeh, A. Azimi, A.K. Iglie, Proximal policy optimization with adaptive generalized advantage estimate, In proceedings of the first international conference on machine learning and knowledge discovery (2024) 453–458.
[7] J. Schulman, S. Levine, P. Abbeel, M. Jordan, P. Moritz, Trust region policy optimization, In International conference on machine learning (2015) 1889–1897.
[8] J. Schulman, P. Moritz, S. Levine, M. Jordan, P. Abbeel, High-dimensional continuous control using generalized advantage estimation (2015) arXiv:1506.02438.
[9] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms (2017) arXiv:1707.06347.
[10] R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, MIT Press, Second edition, 2018.
[11] E. Todorov, T. Erez, Y. Tassa, MuJoCo: A physics engine for model-based control, In 2012 IEEE/RSJ international conference on intelligent robots and systems, IEEE (2012) 5026–5033.
[12] S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, N. Heess, Y. Tassa, dm control: Software and tasks for continuous control, Softw. Impacts 6 (2020) 10022.
Journal of Mathematical Modeling
Volume 14, Issue 1
March 2026
Pages 177-190

Files

Share

How to cite

Statistics