Modifications for Quasi-Newton Method and Its Spectral Algorithm for Solving Unconstrained Optimization Problems

In this paper, two modifications for spectral quasi-Newton algorithm of type BFGS are imposed. In the first algorithm, named SQN EI , a certain spectral parameter is used in such a step for BFGS algorithm differs from other presented algorithms. The second algorithm, SQN Ev-Iv , has both new parameter position and value suggestion. In SQN EI and SQN Ev-Iv methods, the parameters are involved in a search direction after an approximated Hessian matrix is updated. It is provided that two methods are effective under some assumptions. Moreover, the sufficient descent property is proved as well as the global and superlinear convergence for SQN Ev-Iv and SQN EI . Both of them are superior the standard BFGS (QN BFGS ) and previous spectral quasi-Newton (SQN LC ). However, SQN Ev-Iv is outstanding SQN EI if it is convergent to the solution. This means that, two modified methods are in the race for the more efficiency method in terms less iteration numbers and consuming time in running CPU. Finally, numerical results are presented for the four algorithms by running list of test problems with inexact line search satisfying Armijo condition.


Introduction
The optimization problem is a model among many mathematical approaches that deals with solving real life problems, solving exactly and numerically, for different branches; such as, statistics; physics and engineering.This leads to more attempted from the researchers to present more efficient methods continuously.

Now, consider a minimization of unconstrained problem:
min (),  ∈ ℛ 𝑛 1 where, : ℛ  → ℛ is bounded below and twice differentiable function.Many numerical methods are used for solving Eq.1.Quasi-Newton approaches are among the most recommended methods having the efficiency in solving these types of problems, due to their super-linear rate of convergence and global convergent property.
In this most popular algorithm, the inverse of Hessian matrix will be approximated by the formula:  whereas, it begins with initial positive definite matrix  0 up to the required steps.The positive definite property of Hessian matrix approximation or the preserve it after any modifications is the matter that many researchers deal with it.For instance, Mahmood 2 gave a modification BFGS update formula the inverse version, tried to show how it remains symmetric and positive definite, this is the reason to make the problem convergence to the solution in minimization problems.Additionally, this issue is serious in other quasi-Newton method types 3 .The self-correcting property is a technique to overcome the illconditioned problem, for this, Cheng and Li 4 scaled the quasi-Newton equation and suggested a new spectral scaling for BFGS method with this property.Their method has the property of selfcorrecting alike as conventional BFGS has with more efficiency in correcting a large eigenvalue of Hessian matrix approximation might suffer from.This leads to the improvement in BFGS method.Minutely, in their work, they use the exact line search to minimize strictly convex problem from the dimension , in which it is terminated in  steps as steepest descent method.Also, in uniformly convex problems, the method with Wolfe condition is globally and R-linear convergent.Nakayama et al., 5 studied a symmetry rank 1 memoryless quasi-Newton with a parameter of spectral scaling given in 4 .Later, the global convergence of the formula proposed by Nakayama and Narushima 6 .Finally, the hybridization of it with three-term conjugate gradient utilizing a spectral parameter was designed by Nakayama 7 .Additionally, the efficiency of memoryless BFGS method using spectral scaling of 4 in minimizing the eigenvalues was proved by Lv et al. 8 .However, conjugate gradient methods (CGM) have many studies in this area.Firstly, a gradient parameter is used in proposing a new spectral scaling parameter 9 .Another idea was nested between spectral parameter and CGM one and this was given by Wang et al., 10 with presenting the importance of it in solving a large-scale problems.Furthermore, a fast CGM is given with combining a new direction with spectral parameter and previous direction, this is proposed in 11 .Also, the convex combination idea take a place in this topic, the spectral scaling defined as a convex combination of two CGM coefficients 12 .In a constrained optimization problem with bounded condition, there is a spectral parameter with memoryless property for Broydon class presented by Nakayama et al. 13 .Eventually, for a real live example, scientist use the spectral algorithms to analyze the problems as a drug abuse problems; see 14 .After all the presented works, SQN EI and SQN Ev-Iv algorithms are proposed; that deal with new position for spectral scaling parameter and value.In more details, it is an idea to think how changing the involving parameters out of updating formula for Hessian matrix approximation in BFGS method is affecting in the optimization processing.This is aiming at finding more efficient algorithms in spectral scaling methods due to their importance in soling real-life problems.The article, presents two methods with their step algorithms in next section.The third section is proved the convergence of the two method and the relationship between them along with some mild assumptions.There is a proof for the sufficient decent, global and superlinear rate convergence.Finally, the numerical results are presented.

Materials and Methods
In this work, two spectral quasi-Newton methods have been suggested as follows: The SQN Ev-Iv Algorithm In the first part, the SQN Ev-Iv algorithm is suggested, in which the acceleration parameter is involved in https://dx.doi.org/10.21123/bsj.2023.8020P-ISSN: 2078-8665 -E-ISSN: 2411-7986 Baghdad Science Journal search direction; after the Hessian matrix inverse approximation has done.In other words, the search direction contains all required update for BFGS Hessian matrix formula, then it multiples with the spectral scaling parameter.Now, our new suggestion algorithm SQN Ev-Iv ; which the spectral parameter sets in the direction is as following: with, the condition: Thus, the algorithm steps are given as: Step 1: Initializations step: choose  0 identity matrix or positive definite matrix  0 ∈ ℝ  ,tolerance  = 1 × 10 −7 Step 2: Start with  0 = − 0  0 ,  = 1 Step 3: Termination criteria, if ‖  ‖ ≤  or maximum number of iterations reached Stop.

The SQN EI Algorithm
This subsection is about another new algorithm named SQN EI .The spectral parameter of Cheng and Li (2010) 4 is used but in different step of algorithm.The direction is given as this formula: So, the steps of the SQN EI algorithm are the same as previous subsection, but without step 6 and in step 8,   is defined as in Eq.8.
The Convergence Analysis of SQN Ev-Iv and SQN EI The convergence analysis of our algorithm is discussed in this part.

A list of Assumptions:
For conducting the analysis in section 3, some assumptions are needed as follows: (i) Assume that , an objective function, is twice continuously differentiable. (ii) The Lipschitz continuous properties for the Hessian matrix at  * , that is, it is satisfying the inequality: with the existing of a positive constant c and all  in neighborhood of  * (iii) If the objective function  ∈  2 and  = {: () ≤ ( 0 )} is a convex level set, then there exist two positives  1 and  2 satisfying  1 ‖‖ 2 ≤  ′ ∇ 2 () ≤  2 ‖‖ 2 ∀  ∈ ℜ  ,  ∈  and ∇ 2 () is Hessian matrix of .

The Relationship between SQN Ev-Iv and SQN EI Parameters
It is obvious that, SQN Ev-Iv parameter is a function of SQN EI .However, there is a relation between them; as long as there is a condition for the one in SQN Ev-Iv , this means that: Then, multiplying both sides of Eq.9 ‖  ‖ 2 , and taking the square root to them to obtain the following: Which means the parameter of SQN Ev-Iv is less than the parameter of SQN EI .

Sufficient Descent Property of Two Algorithms
In this algorithm, assumed that the direction as given in Eq.6 and Eq.8, that is; it is used to prove the descent direction: This means that it has descent direction property for  = 0. Now, it is wanted to prove for  ≥ 1.
Since, in this part the direction search is given as Therefore,      has descent direction property in SQN Ev-Iv .In the same way for SQN EI the property holds for it is gotten.

The Global Convergence Analysis
In order to prove the global convergence of the proposed algorithms, Lemma 7 in 15 showed a line search with Armijo condition and the property of descent direction satisfies one or both following inequalities.That is, if ℎ( +1 ) = ( +1 ) − (  ), then either: since   is bounded then ‖  ‖ < , therefore: and by assumption 1 (iii), it is: where,  and  are positive constant.
Remark 1: Theorem 1 is hold, the global convergence, when the direction of SQN EI algorithm is used.Therefore, by subsection 2, the same result is obtained.

Superlinear Rate of Convergence
In this section, the superlinear convergence is presented and proved.In order to do that, some principles are need and recall them in 16 two lemmas 4.9 and 4.10.Then with holding the assumption 1, a sequence of numbers as {  } such that whenever,   and   are given in Eq.5.These all tend to the boundedness of the sequence of Hessian matrix approximation and its inverse, that is, {  } and {  −1 }: Therefore: Proof: Therefore: as it has given that: On the other hand:

Results and Discussion
This section gives the results and presents all findings throughout the performance profile plots, the cumulative distribution function.In this way, the significant difference will only show in the interesting area.The time of CPU running and the number of iterations are two criterions in showing the effectiveness of the suggested algorithms in numerical optimization branch.This way is used in this paper.For the comparison, 46 function tests are used, that are in each of 17 and 18 , as listed in Table 1.About the dimensions, the various dimensions for functions are taken when the function is multivariate.This means that, if  is the dimension number, then for the first three functions in Table 1, when ID = 1, 2 and 3,  = 2 is used, while for all other listed functions;  = 2, 4, 6, 8, 15,30 is used except Diagonal 2, ID = 14, the used dinamsions were  = 2,3,4,10,15.As a final result, the overall data becomes 260 for plotting the performance profile.The Fig. 1 presents plots for the four procedures.In details, Fig. 1(a) shows how the SQN Ev-Iv algorithm behaves in terms of iteration numbers for all test functions.The SQN EI is preferable than QN BFGS and SQN LC methods for reducing iteration numbers.However, this result is inaccurate with SQN Ev-Iv algorithm after satisfying the condition Eq.7.
Whenever the problem is convergent, SQN Ev-Iv is more recommended among the four algorithms.Meanwhile, Fig. 1(b) reveals the time consuming of CPU for running all of contest algorithms.Again, the SQN Ev-Iv procedure is better than others.For conducting the analysis in Fig. 1, the number of test functions was 43; that is, all functions identify in Table 1 is utlized excluding functions  = 24,37, 39.Furthermore, there were two functions,  = 5, 11, made a terrible for some dimension, for instance, function with  = 5,  = 6, 20, 30 and  = 11,  = 20, 30 are excluded from the analysis.In other word, 43 test problems are filtered with Eq.7.
On the other hand, Fig. 2 demonstrates how the cumulative distribution line changes when involves all 46 problems named in Table 1; but inequality Eq.7 is ignored in SQN Ev-Iv algorithm.Along with this failure, SQN Ev-Iv remains the selected algorithm comparing with QN BFGS and SQN LC except SQN EI .Overall, SQN EI is dominates the all engaged methods in this study without the inequality Eq.7 holds.For all programs, the MATLAB 2018a codes are written with using inexact line search satisfying the strong Wolf condition and the error  = 1 × 10 −7 or iteration number reached at maximum number, in which it was 1000; where  is the dimension of objective function.Furthermore, the Table 1 contains the name of all functions used in the comparison with some suggested dimensions.For running programs, the suggested initial values for those functions given in 17 ; is used however, there were some testing functions with no initial points; in this case, the border values of defined region is used as an initial point in 18 .

Conclusion
The paper suggested two new algorithms, SQN EI and SQN Ev-Iv .SQN EI depended on the new position in the usage of defined spectral parameter for the past work while, SQN Ev-Iv beside the new place of it; there is a new value to accelerate the process of solving problems.In the past, one or two spectral parameters were used in a Hessian approximation matrix updating formula.However, this new technique shows the effectiveness of approaches in optimizing problems.That is to say, SQN EI and SQN Ev-Iv algorithms were preferable in comparison to each of QN BFGS and SQN LC according to running computer system processer and iteration numbers.In general, SQN EI is better than others.However, there is a contest between the two proposed algorithms by a condition decision; it made SQN Ev- =   −  −1 ,   =   −  −1 5

Figure 1 .Figure 2 .
Performance profile for four algorithms, with filtered functions (a) Number of iterations and (Performance profile for algorithms, with all functions (a) Number of iterations and (b) CPU running time + (  +    ) ′