A new version of augmented self-scaling BFGS method

Document Type : Research Article

Authors

1 Department of Mathematics, Payame Noor University, P.O. Box. 19395-3697, Tehran, Iran

2 Department of Mathematics, Technical and Vocational University (TVU), Tehran, Iran

Abstract

A new version of the augmented self-scaling memoryless BFGS quasi-Newton update,  proposed in [Appl. Numer. Math. 167,  187--201,  (2021)],  is suggested for unconstrained optimization problems. To use the corresponding scaled parameter,  the clustering of the eigenvalues of the approximate Hessian matrix about one point is applied with three approaches. The first and second approaches are based on the trace and the determinant of the matrix. The third approach is based on minimizing the measure function. The sufficient descent property is guaranteed for uniformly convex functions,  and the global convergence of the proposed algorithm is proved both for the uniformly convex and general nonlinear objective functions,  separately. Numerical experiments on a set of test functions of the CUTEr collection show that the proposed method is robust. In addition,  the proposed algorithm is effectively applied to the salt and pepper noise elimination problem.

Keywords

Main Subjects


[1] Z. Aminifard, S. Babaie-Kafaki, Dai-Liao extensions of a descent hybrid nonlinear conjugate gra-
dient method with application in signal processing, Numer. Algorithms 89 (2021) 1369–1387.
[2] Z. Aminifard, S. Babaie-Kafaki, S. Ghafoori, An augmented memoryless BFGS method based on a
modified secant equation with application to compressed sensing, Appl. Numer. Math. 167 (2021)
187–201.
[3] N. Andrei, Eigenvalues versus singular values study in conjugate gradient algorithms for large-
scale unconstrained optimization, Optim. Methods Softw. 32 (2017) 534–551.
[4] N. Andrei, Diagonal Approximation of the Hessian by Finite Differences for Unconstrained Opti-
mization, J. Optim. Theory Appl. 185 (2020) 859–879.
[5] N. Andrei, New conjugate gradient algorithms based on self-scaling memoryless Broyden-Fletcher-
Goldfarb-Shanno method, Calcolo 57 (2020) 1–27.
[6] N. Andrei, A double parameter self-scaling memoryless BFGS method for unconstrained optimiza-
tion, Comput. Appl. Math. 39 (2020) 159.
[7] M.R. Arazm, S. Babaie-Kafaki, R. Ghanbari, An extended Dai-Liao conjugate gradient method
with global convergence for nonconvex functions, Matematicki 52 (2017) 361–375.
[8] S. Babaie-Kafaki, A modified scaled memoryless BFGS preconditioned conjugate gradient method
for unconstrained optimization, 4OR 11 (2013) 361–374.
[9] S. Babaie Kafaki, A modified scaling parameter for the memoryless BFGS updating formula, Nu-
mer. Algorithms 72 (2016) 425–433.
[10] S. Babaie-Kafaki, Z. Aminifard, Two parameter scaled memoryless BFGS methods with a non-
monotone choice for the initial step length, Numer. Algorithms 82 (2019) 1345–1357.
[11] S. Babaie-Kafaki, Z. Aminifard, S. Ghafoori, A hybrid quasi-Newton method with application in
sparse recovery, Comput. Appl. Math. 41 (2022) 249.
[12] S. Babaie-Kafaki, N. Mirhoseini, Z. Aminifard, A descent extension of a modified Polak–Ribire–
Polyak method with application in image restoration problem, Optim. Lett. 17 (2023) 351–367.
[13] C.G. Broyden, A class of methods for solving nonlinear simultaneous equations, Math. Comput. 19
(1965) 577–593.
[14] C.G. Broyden, The convergence of a class of double-rank minimization algorithms: 2. the new
algorithm, IMA J. Appl. Math. 6 (1970) 222–231.
[15] R.H. Byrd, J. Nocedal, A tool for the analysis of quasi-Newton methods with application to uncon-
strained minimization, SIAM J. Numer. Anal. 26 (1989) 727–739.
[16] R.H. Byrd, J. Nocedal, Y.-X. Yuan, Global convergence of a class of Quasi-Newton methods on
convex problems, SIAM J. Numer. Anal. 24 (1987) 1171–1190.
[17] Y. H. Dai, C.-X. Kou, A nonlinear conjugate gradient algorithm with an optimal property and an
improved Wolfe line search, SIAM J. Optim. 23 (2013) 296–320.
[18] J.E. Dennis, J.J. More, A characterization of superlinear convergence and its application to quasi-
Newton methods, Math. Comput. 28 (1974) 549–560.
[19] J.E. Dennis, J.J. More, Quasi-Newton methods, motivation and theory, SIAM Review 19 (1977)
46–89.
[20] J.E. Dennis, H. Wolkowicz, Sizing and least-change secant methods, SIAM J. Numer. Anal. 30
(1993).
[21] E.D. Dolan , J.J. More, Benchmarking optimization software with performance profiles, Math. Pro-
gram. 91 (2002) 201–213.
[22] L. Exl, J. Fischbacher, H. Oezelt, M. Gusenbauer, T. Schrefl, Preconditioned nonlinear conjugate
gradient method for micromagnetic energy minimization, Comput. Phys Commun. 235 (2019) 179–
186.
[23] N.I. Gould, D. Orban, P.L. Toint, Cuter and sifdec: A constrained and unconstrained testing envi-
ronment, revisited, ACM Trans. Math. Softw. 29 (2003) 373–394.
[24] A. R. Heravi, G. Abed Hodtani, A New Correntropy-Based Conjugate Gradient Backpropagation
Algorithm for Improving Training in Neural Networks, IEEE Trans. Neural Netw. Learn. Syst. 29
(2018) 6252–6263.
[25] D. H. Li, M. Fukushima, A modified BFGS method and its global convergence in nonconvex mini-
mization, J. Comput. Appl. Math. 129 (2001) 15–35.
[26] M. Li, H. Liu, Z. Liu, A new family of conjugate gradient methods for unconstrained optimization,
J. Appl. Math. Comput. 58 (2018) 219–234.
[27] W. Li, Y. Liu, J. Yang, W. Wu, A new conjugate gradient method with smoothing L1/2 regularization
based on a modified secant equation for training neural networks, Neural Process. Lett. 48 (2018)
955–978.
[28] A. Liao, Modifying the BFGS method, Oper. Res. Lett. 20 (1997) 171–177.
[29] J. Lin, C. Jiang, An improved conjugate gradient parametric detection based on space-time scan,
Signal Process. 169 (2020) 107412.
[30] I.E. Livieris, V. Tampakas, P. Pintelas, A descent hybrid conjugate gradient method based on the
memoryless BFGS update, Numer. Algorithms 79 (2018) 1169–1185.
[31] S. Nezhadhossein, New nonlinear conjugate gradient methods based on optimal Dai-Liao param-
eters, J. Math. Model. 8 (2020) 21–39.
[32] S. Nezhadhossein, A modified descent spectral conjugate gradient method for unconstrained opti-
mization, Iran. J. Sci. Technol. Trans. A Sci. 45 (2021) 209–220.
[33] J. Nocedal, S. J. Wright, Numerical Optimization, Springer Series in Operations Research.
Springer-Verlag, New York, 1999.
[34] S.S. Oren, D.G. Luenberger, Self-scaling variable metric (ssvm) algorithms: Part i: Criteria and
sufficient conditions for scaling a class of algorithms, Manag. Sci. 20 (19774) 845–862.
[35] S.S. Oren, E. Spedicato, Optimal conditioning of self-scaling variable metric algorithms, Math.
Program. 10 (1976) 70–90.
[36] K. Sugiki, Y. Narushima, H. Yabe, Globally convergent three-term conjugate gradient methods
that use secant conditions and generate descent search directions for unconstrained optimization,
J. Optim. Theory Appl. 153 (2012) 733–757.
[37] W. Sun, Y.-X. Yuan, Optimization Theory and Methods: Nonlinear Programming, Springer, New
York, 2006.
[38] N. Ullah, J. Sabiu, A. Shah, A derivative-free scaling memoryless Broyden-Fletcher-Goldfarb-
Shanno method for solving a system of monotone nonlinear equations, Numer. Linear Algebra
Appl. 28 (2021) e2374.
[39] Z. Wei, G. Li, L. Qi, New quasi-Newton methods for unconstrained optimization problems, Appl.
Math. Comput. 175 (2006) 1156–1188.
[40] R. Winther, Some superlinear convergence results for the conjugate gradient method, SIAM J.
Numer. Anal. 17 (1980) 14–17.
[41] G. Yu, J. Huang, Y. Zhou, A descent spectral conjugate gradient method for impulse noise removal,
Appl. Math. Lett. 23 (2010) 555–560.
[42] G. Yuan, J. Lu, Z. Wang The PRP conjugate gradient algorithm with a modified WWP line search
and its application in the image restoration problems, Appl. Numer. Math. 152 (2020) 1–11.
[43] J.Z. Zhang, N.Y. Deng, L.H. Chen, New quasi-Newton equation and related methods for uncon-
strained optimization, J. Optim. Theory Appl. 102 (1999) 147–167.
[44] W. Zhou, L. Zhang, A nonlinear conjugate gradient method based on the MBFGS secant condition,
Optim. Methods Softw. 21 (2006) 707–714.