Unifying The Evaluation Criteria Of Many Objectives Optimization Using Fuzzy Delphi Method

. Abstract: Many objective optimizations (MaOO) algorithms that intends to solve problems with many objectives (MaOP) (i.e., the problem with more than three objectives) are widely used in various areas such as industrial manufacturing, transportation, sustainability, and even in the medical sector. Various approaches of MaOO algorithms are available and employed to handle different MaOP cases. In contrast, the performance of the MaOO algorithms assesses based on the balance between the convergence and diversity of the non-dominated solutions measured using different evaluation criteria of the quality performance indicators. Although many evaluation criteria are available, yet most of the evaluation and benchmarking of the MaOO with state-of-art algorithms perform using one or two performance indicators without clear evidence or justification of the efficiency of these indicators over others. Thus, unify a set of most suitable evaluation criteria of the MaOO is needed. This study proposed a distinct unifying model for the MaOO evaluation criteria using the fuzzy Delphi method. The study followed a systematic procedure to analyze 49 evaluation criteria, sub-criteria, and its performance indicators, a penal of 23 domain experts, participated in this study. Lastly, the most suitable criteria outcomes are formulated in the unifying model and evaluate by experts to verify the appropriateness and suitability of the model in assessing the MaOO algorithms fairly and effectively.


Introduction:
In many-objective optimization algorithms (MaOO), the performance evaluation considers a critical matter in determining the accuracy of the results 1, 2 . Many evaluation criteria were proposed in the context of MaOO to evaluate the MaOO algorithms.
However, MaOO evaluation performance considers a primary challenge in the optimization process due to the complexity of MaOPs and the ambiguity of criteria selection that depend mainly on one or two of the evaluation criteria metrics 3,4 . Despite the fact that some of these criteria have been criticised in the literature, they are still chosen at random and used to evaluate the performance of MaOO algorithms. In addition, the process of selecting any of these criteria and its metrics for evaluation remains an open question 5,6 . This study aims to unify the most suitable evaluation criteria for evaluating the MaOO performance using the fuzzy Delphi method.
Fuzzy Delphi Method (FDM), proposed based on integrating the Delphi method into fuzzy theory 7 to overcome the Delphi method drawbacks. FDM has proven effectiveness in widely employed for unifying, screening and forecasting, assessment, standardization, and criteria identification in various domains [7][8][9][10][11][12][13] . Mainly, FDM provides appropriate results when making decisions regarding objective issues needed, while the involving criteria are not unified 13,14 .
The FDM provides a resilient framework that can handle the lack of precision and clarity. The incomplete or inaccurate information is considered an issue in decision making.
Furthermore, subjectivity in the decisions made by the decision-makers caused uncertain results. FDM is tailored to the fuzzy environment to handle imprecise descriptions and human subjectivity 12 . Employing fuzzy numbers can leave the impression of using an appropriate method for decision-making 8 . Therefore, FDM is the best method for assessing and unifying the most effective criteria with high flexibility scale 8,9,13 . Moreover, all vital information will be considered without any missing because the membership degree effectively considers all experts' preferences 9,14,15 .

Methodology:
The process for unifying the evaluation criteria model in the context of MaOO is introduced in this section. The process comprises three steps, as shown in Figure 1, which are: (1) MaOO criteria identification and analysis, (2)   Based on Figure 1, the first step for unifying the MaOO criteria is collecting, identifying, and analysing all MaOO evaluation criteria to criticize and categorize them based on their group. According to study 16 there are 49 criteria, subcriteria, and indicators are used and developed to assess the performance of the MaOO algorithms, which are classified and presented as will present in the results section; Second step is Fuzzy Delphi Analysis (FDM) which consists of five stages. Stage 1: First, the panelist of experts from the domain are selected, while the recommended number of experts range is between (10)(11)(12)(13)(14)(15) 17 ; this range is sufficient when their backgrounds are the same, and uniformity is high. However, some studies included 50 specialists 18 . Stage 2: formulate the expert evaluation form (questionnaire) to collect the expert consensus over the studied criteria using the five Likert scale 19,20 . Stage 3: Data dissemination and data collection: the experts' feedback was collected through the developed questionnaire using an online survey 21 . Stage 4: the collected data of expert evaluation per criteria convert from linguistic scale to triangular fuzzy number using table 1. Lastly, stage 5: in this step, the degree of agreement for each criterion based on expert consensus are computed. Three acceptance conditions are employed, which are, a) the results of applying vertex method to find d value for each criterion should be less or equal to 0.2 22 ; b) the percentage consensus for each sub-criterion and overall should be greater than 75% 23 ; and c) the average of the fuzzy number for each criterion should be greater than ⍺-cut of 0.5 value. This value supports the sensible reasoning that only those elements from the fuzzy support set with 'sufficiently large' group grades are included 24 . Consequently, the results of the criterion must pass all three conditions 13,14 , then this criterion gains the expert consensus and will be included in the unified model; otherwise, it will be omitted; Lastly, the validation step as demonstrated in Figure 1.

Results and Discussion:
Many performance indicators proposed to evaluate objective optimization algorithms. Some of them gradually decayed as it's inefficient to work with more than three objectives, while many others are developing to assess MaOO algorithms' performance. However, there is still an issue in evaluating the MaOO algorithms when it's come to select the evaluation criteria for benchmarking or assessing the MaOO algorithms, as there is no clear evidence of the reason behind using specific one or two indicators amongst others. Thus, proposing an evaluation criteria model is a necessity that aims to unify the most suitable criteria for MaOO. As present in the methodology section, the process of developing an evaluation criteria model comprises three steps: (1) Identify the evaluation criteria for MaOO: In this section, all criteria, sub-criteria, and indicators are collected, combined, and categorized from literature. A summary of them recalled and listed down here as shown in Table 1.
(2) fuzzy Delphi analysis: As mentioned in methodology section, the fuzzy Delphi method was employed to analyze the experts' consensus on "MaOO evaluation criteria" to identify the most suitable criteria. This step started by developing a questionnaire for fuzzy Delphi analysis as an instrument for collecting expert opinions over the 49 criteria, sub-criteria and its indicators identified, collected and analysed in the previous step and ending with testing the acceptance conditions on the output criteria set. The processing results of these steps are presented in detail as following: Step 1: Expert selection is a critical task: For expert selection, the non-probability purposive sampling was used based on the following inclusion criteria: (a) MaOO developer specialist either industrial or academician (b) Those who have possessed a remarkable research work in the field of MaOO. (c) Minimum 5 years of experience in the study field. About 250 experts from around the world contacted and invited to participate through LinkedIn or their official email address; 50 of them expressed their readiness for participating in this research study.
Step 2: Developing the expert questionnaire; Parallelly, the fuzzy Delphi analysis questionnaire formulated for the 49 criteria, sub-criteria, and indicators items. The face and content validity were utilized to check the validity and reliability level of the expert questionnaire before conducting the actual study. The content validity index (CVI) results were between 0.857 (6/7) to 1 (7/7), shows that all questionnaire items are relevant and valid. While the average item level (S-CVI/Ave) and Universal Agreement (UA) among experts (S-CVI/UA) were 0.994 and 0.960, respectively. Thus, the developed instrument of this study is valid 25,26 . The first version of the fuzzy Delphi analysis questionnaire was sent for a pilot study to test its reliability. Twenty participants from University Putra Malaysia (UPM) and the University of Baghdad were invited to answer the survey. The collected responses were analyzed using SPSS to compute the reliability level. The accepted Cronbach's alpha is 0.75, the Cronbach's alpha for the fuzzy Delphi instrument was 0.944.  Step 3: Survey dissemination and data collection; The online survey was distributed using email tool and weblink on Smartsurvey.com and shared with the experts from around the world, who were instructed to use a five Likert scale to express their agreement level for each criterion, subcriterion, and indicator. Twenty-three experts submitted a completed response, the response data exported from the questionnaire as an input to analyse in fuzzy Delphi method.
Step 4: In this step, all experts' collected results are converted into fuzzy triangular numbering from the linguistic variables as shown in Table 2, the process of converting the main criteria data to fuzzy numbers. The exact process applied to the rest of the subcriteria and indicators. Strongly Agree (0.6,0.8,1.0) Step 5: The last step is testing the acceptance conditions for each item (i.e., criteria, subcriteria, and indicators). Table 3 shows the results of the three conditions representing the experts' consensus on the main criteria, the same process applied to the rest of the subcriteria and indicators.  As shown in Table 3, the cardinality criterion passed the first two conditions, but the third condition didn't pass, in conclude, this criterion is rejected, and consequently, the indicators of this criterion will be rejected, as well. The unified model of MaOO evaluation criteria, subcriteria, and indicators analysis results based on fuzzy Delphi analysis is demonstrated in Table 4. Table 4 demonstrates the unified model result based on the fuzzy Delphi process of the acceptance conditions for all tested criteria, subcriteria, and indicators. Out of three main criteria, two criteria passed all three conditions namely the Pareto and Time, while the cardinality criterion rejected. For time criterion's indicators, two indicators are accepted, which are ARE and Performance score. For Pareto criterion, all other subcriteria with single and multi-directions got the experts' consensus on its suitability for evaluating the MaOO algorithms. However, the accepted indicators per each subcriterion were as following; For convergence criterion out of eight indicators, six indicators were accepted: generation distance, LGD, the additive epsilon, the power mean of generational distance, pertinence metric, and convergence metric.  As shown in Figure 2, the experts have been asked to validate the proposed model and show their level of acceptance, and the results of their responses support for the proposed model and its validity, out of 21 experts who participated in the validation process (14; 66.67%) show their acceptance and (6; 28.57%) was strongly accepted, while 4.76% (1 expert) request to revise the model and replace the abbreviation or acronym name with the full name and the researcher already applied that, and the final approved model presented in Table 4.

Conclusion:
The Many Objective Optimization algorithms (MaOO) evaluation criteria play a critical role in evaluating the competition MaOO algorithms. Although these criteria have been criticized in literature, they are employed in the evaluation randomly, and the process of selecting them remains unclear. Thus, the need for unifying the criteria set became inevitable. This study presents the processing results of developing criteria model for many objectives optimization algorithms. The fuzzy Delphi analysis test and refine the 49 criteria, subcriteria, and its indicators. The fuzzy Delphi method's final results narrowed the criteria set down to 31 of the most suitable criteria set. The statistical analysis of the experts' evaluation proved the validity of the proposed criteria model and its suitability for evaluating the MaOO algorithms. These results contribute to the body of knowledge and provide a flexible unified model of evaluation criteria for MaOO algorithms. In future work and to provide an exhaustive evaluation methodology, the importance level of each of these suitable criteria set needs to study and determine accordingly.