Dynamic adaptation strategies for optimal control in unknown linear time-invariant system

Document Type : Research Article

Authors

Department of Applied Mathematics, Faculty of Mathematical Sciences, Ferdowsi University of Mashhad, P. O. Box 1159, Mashhad 91775, Iran

Abstract

This paper presents a framework for online adaptive optimal control of continuous-time linear systems with unknown dynamics. The approach uses approximate and adaptive dynamic programming to learn the optimal control policy and value function in real-time, without prior knowledge of the system matrices. We introduce two algorithms based on policy iteration and value iteration, providing proofs the convergence and stability. Our value iteration method is robust against  from  exploration noise. The effectiveness of these control strategies is demonstrated through two examples, highlighting their ability to achieve near-optimal performance despite unknown dynamics. 

Keywords

Main Subjects

References

[1] T. Bian, Z.P. Jiang, Value iteration and adaptive dynamic programming for data-driven adaptive optimal control design, Automatica 71 (2016) 348–360.
[2] S. Effati, M. Pakdaman, Optimal control problem via neural networks, Neural Comput. Appl. 23 (2013) 2093–2100.
[3] S. Effati, A. Mansoori, M. Eshaghnezhad, Linear quadratic optimal control problem with fuzzy variables via neural network, J. Exp. Theor. Artif. Intell. 33 (2021) 283–296.
[4] W. Gao, M. Mynuddin, D. C. Wunsch, Z. P. Jiang, Reinforcement learning-based cooperative optimal output regulation via distributed adaptive internal model, IEEE Trans. Neural Netw. Learn. Syst. 33 (2021) 5229–5240.
[5] P.A. Ioannou, J. Sun, Robust Adaptive Control, PTR Prentice-Hall, Upper Saddle River, NJ, 1996.
[6] Y. Jiang, Z.P. Jiang, Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics, Automatica. 48 (2012) 2699–2704.
[7] M. Jung, K. Glover, U. Christen, Comparison of uncertainty parameterisations for H∞ robust control of turbocharged diesel engines, Control Eng. Pract. 13 (2005) 15–25.
[8] G. L. Kharatishvili, The maximum principle in the theory of optimal processes involving delay, Dokl. Akad. Nauk. 136 (1961) 39–42.
[9] D. Kleinman, On an iterative technique for Riccati equation computations, IEEE Trans. Automat. Control. 13 (1968) 114–115.
[10] A.J. Laub, Matrix Analysis for Scientists and Engineers, Society for Industrial and Applied Mathematics, Philadelphia, 2004.
[11] F. Lewis, D. Vrabie, Reinforcement learning and adaptive dynamic programming for feedback control, IEEE Circuits Syst. Mag. 9 (2009) 32–50.
[12] F. Lewis, D. Vrabie, V.L. Syrmos, Optimal Control, John Wiley & Sons, 2012.
[13] Y.S. Ma, J. Sun, Y. Xu, S.S. Cui, Z.G. Wu, Adaptive dynamic programming for optimal control of unknown LTI system via interval excitation, IEEE Trans. Automat. Control. 70(7) (2025) 4896–4903. Online published, https://doi.org/10.1109/TAC.2025.3542328.
[14] A. Mansoori, S. Effati, M. Eshaghnezhad, An iterative method to solve the time delay optimal control problem with bounded control variable, Int. J. Comput. Math. 101 (2024) 653–667.
[15] H. Modares, F. Lewis, Z. P. Jiang, Optimal output-feedback control of unknown continuous-time linear systems using off-policy reinforcement learning, IEEE Trans. Cybern. 46 (2016) 2401–2410.
[16] D.S. Naidu, Optimal Control Systems, CRC press, 2018.
[17] L.S. Pontryagin, Mathematical Theory of Optimal Processes, Routledge, 2018.
[18] S.A. Rizvi, Z. Lin, Reinforcement learning-based linear quadratic regulation of continuous-time systems using dynamic output feedback, IEEE Trans. Cybern. 50 (2019) 4670–4679.
[19] B.L. Stevens, F. Lewis, E.N. Johnson, Aircraft Control and Simulation: Dynamics, Controls Design, and Autonomous Systems, John Wiley & Sons, 2015.
[20] R.S. Sutton, A.G. Barto, Reinforcement Learning: An Introduction, Bradford Book, Cambridge, MA, 2018.
[21] G. Tao, Adaptive Control Design and Analysis, John Wiley & Sons, 2003.
[22] D. Vrabie, O. Pastravanu, M. Abu-Khalaf, F. Lewis, Adaptive optimal control for continuous-time linear systems based on policy iteration, Automatica 45 (2009) 477–484.
[23] K. Xie, Y. Zheng, W. Lan, X. Yu, Adaptive optimal output regulation of unknown linear continuous-time systems by dynamic output feedback and value iteration, Control Eng. Pract. 141 (2023) 105675.
[24] M. Zarefard, S. Effati, Adaptive synchronization between two non-identical BAM neural networks with unknown parameters and time-varying delay, Int. J. Control Autom. Syst. 15 (2017) 1877–1887.
Journal of Mathematical Modeling
Volume 14, Issue 1
March 2026
Pages 1-17

Files

Share

How to cite

Statistics