CTJ: Input-Output Based Relation Combinatorial Testing Strategy Using Jaya Algorithm

: Software testing is a vital part of the software development life cycle. In many cases, the system under test has more than one input making the testing efforts for every exhaustive combination impossible (i


Introduction:
Combinatorial testing is a black-box testing technique that generates test cases by combining the values of different input parameters using combinatorial optimization strategies to reduce the interfaces fail and increase the reliability of the system (1). Taking the study from the failure of the medical device application, the failure-triggering fault interaction (FTFI) is 68% for the single parameter value, 97% of failures triggered by 2 combination values while the percentage of failures caused by 3 and 4 combination values are 99% and 100% respectively (2). By using combinatorial testing, all input values of the test objects and interactions between each parameter are tested. This testing causes a higher detection of interaction failures compared to single parameter testing.
More specifically, combinatorial test generation relates to the process of searching the optimum number of test cases for test consideration based on the interaction of t-way parameters (where t indicates the interaction strength). Many different optimization strategies are used to generate the test cases for combinatorial testing such as Harmony Search (3), Genetic Algorithm (4), Ant Colony Algorithm (4), Simplified Swarm Optimization (5), Differential Evolution Algorithm (6) and so on.
Most of these aforementioned algorithms have control parameters. For example, Harmony Search requires 4 parameter controls namely maximum iteration, harmony size, harmony memory consideration rate, and pitch adjustment. Similarly, Ant Colony adopts maximum iteration, population size, evaporation rate, pheromone influence, and heuristic influence as the parameter controls. Tuning of these control parameters is necessary to ensure the balance between exploration (i.e. sufficiently roaming through all the potential areas in the search space) and exploitation (i.e. searching around the known best). Often, the tuning of these control parameters is difficult as poor tuning can lead to falling into local optima and unnecessary increase in computational performance. Addressing these issues, our work proposes to adopt the Jaya algorithm as the backbone algorithm for a combinatorial optimization problem. Apart from being a significantly new algorithm, the Jaya algorithm (7) has only two parameter controls (i.e. common controls to all search algorithms: population size and maximum iteration). This is an important feature as the Jaya algorithm offers a fair learning curve as well as straightforward adoption effort as there is no need for significant tuning (8,9). Additionally, the ease of resolving discrete optimization problems and convergence to global optimum value make the Jaya algorithm even better than other optimization algorithms (10). Furthermore, the Jaya algorithm offers a faster convergence speed than that of the TLBO algorithm. Hence, a new strategy in overcoming input-output based relation combinatorial testing using Jaya algorithm called CTJ is introduced. Summing up, the main contribution of the work can be summarized as follows:  Unlike the state-of-the-art implementation, CTJ is the first implementation that adopts parameterfree algorithm as the backbone algorithm.  CTJ also supports combinatorial input-output based relation.  CTJ performs well when compared with the state-of-the-art implementations.

Related Work:
This section contains the mathematical notation of Input-output based relation combinatorial testing, followed by a survey on combinatorial testing. To express IOR in a mathematical form, a combination of input-output relationship (Rel) and covering array is needed to come out with inputoutput based relations covering array. Rel can be written in this form, Rel = {{x1}, {x2}, … ,{xn}} where x is the combination of inputs that will generate the specific output. Input-output based relations covering array, IOR (N, C, Rel) is the mathematical form of IOR relation. N represents the number of test case in the test suite, C is the number of value of each input parameter (v1P1, v2P2, … , vnPn) where v is the amount of input value and p is the amount of parameter that has the same amount of v while Rel is the input-output relationship as stated above (11).
The greedy algorithm has previously been used by Schroeder, the researcher who proposed combinatorial testing with IOR feature in one of his studies to determine particular combinations of inputs which affect the outputs of the program (12). Furthermore, the study of the variable strength combinatorial test suite using the Greedy algorithm is done by Wang and his colleagues to increase the flexibility of controlling the interaction strength (13).
The density-based algorithm was first applied in the research in interaction testing by Colbourn and his associates, but it was limited only to pairwise testing (14). Colbourn and his colleagues then continued the research by increasing the limit of interaction strength (15). There is another research conducted by Wang and his colleagues that introduced variable strength interaction in combinatorial testing using density as the optimization algorithm (13).
AURA is a non-deterministic input-output based relationship combinatorial testing strategy proposed by Ong and Kamal (16). This strategy is focusing on solving the mapping of symbolic values to actual data manually and the lack of flexibility of existing test suite generation. AURA supports uniform interaction strength and variable interaction strength with the strength up to 3 as well as input-output based relationships.
There are several types of research have been done on showing the implementation of the Ant Colony Optimization algorithm in combinatorial testing. These include the study of comparing the efficiency of ACO with simulated annealing and genetic algorithm (17) as well as applying ACO in variable interaction strength combinatorial testing (18). ACO has been proposed to be applied in IOR combinatorial testing in recent research by Ramli and her associates (19) but there is a lack of results of implementation in the study.
Genetic Algorithm has been implemented in solving combinatorial testing optimization problems such as in the research by Shiba and his associates (4). Srivastava and Kim developed variable strength interaction combinatorial testing using GA to focus on the parts that are critical by implementing a more selective approach (20). Furthermore, McCaffrey conducted a study to identify the effectiveness of GA in pairwise testing  (21). Nevertheless, GA has not yet been applied for the optimization in combinatorial testing that has an IOR feature. Several types of research have applied the Harmony Search algorithm in combinatorial testing. In 2011, Alsewari applied the HS algorithm in t-way interaction test data generation (22). Besides, the HS algorithm is being implemented in pairwise testing strategy (PHSS) and PHSS is outperformed existing strategies in terms of the size of the test suite generated in the study in (23). Besides, utilization of HS algorithm in variable strength interaction combinatorial testing with constraint support is carried out by (3). However, to the best of the authors' knowledge, there is no research deals with the IOR feature using HS.
Based on the findings, Particle Swarm Optimization has been implemented in pairwise testing by Chinese researchers in 2010 (24). Other than that, test suite generation using variable interaction strength also implemented PSO to solve the combinatorial problem (25). Still, there is no combinatorial testing that employs the IOR feature using PSO.
There are few types of research related to combinatorial testing using Simulated Annealing for optimization that have been carried out. Cohen and Colbourn used SA to solve the optimization problem while constructing a test suite for interaction testing (26). SA also is used to combine with algebraic construction to build covering arrays for interaction testing that is strength three (27). Again, no research uses the SA algorithm to optimize the combinatorial problem in combinatorial testing that features IOR.

CTJ Implementation:
This section gives a details description of the proposed Jaya algorithm. To realize the inputoutput based relation combinatorial testing based on Jaya algorithm, there are five levels of actions to be carried out. The five-level actions start by reading the input values entered by the user, data analyzation and data mapping, input values combination generation, test case generation, and final test suite generation.

Step1: Data Analysis and Data Mapping
The purpose of the analysis is to ensure the information user keyed in is in the right format and syntax. Any wrong information inclusive of amiss format and syntax entered will cause the system not able to recognize even more the system will crash. Therefore, preventive action is taken to counter the happening of the above situation by giving users feedback messages so that they can recheck the problem and make the correction.
Data mapping is carried out right after data analysis. The input values from each parameter that has been verified during the data analysis process will undergo the mapping process with integers. By applying data mapping, the time taken to generate all possible combinations of input values and the test case will be reduced due to the size of the input data is decreased. Often, the size of a string is larger than an integer. Smaller byte of data always processes faster than the larger one. Hence, the string values of input data are being substituted with integers during data processing.

Step2: Combinations of Input Value Generation
In this step, the combination of input values is generated to be used in test case generation based on the interaction strength or based on the selection input/output combinations. Each input value that belongs to the same parameter and has mapped to the corresponding integer is merged with other values of parameters to form combinations of values. There are two types of combinations implemented which are combinations of input values based on input-output relationship and interaction strength.

Step3: Test Case Generation Based on Jaya Algorithm
After gathering all combinations of input values, the next step is to generate the test case. It starts with generating a test case by randomly pick one of the input values from each parameter. The generated test case which will be assessed by determining the number of combinations of input values that generated in Step2 covered by the test case. The best and worst test cases in terms of coverage in the population will be picked for modification purposes. Each test case in the population will be improved by applying the modification formula in the Jaya algorithm (Eq. (1)) based on the best and worst test cases. If the test case generated after employing the modification has better coverage than the previous one, it will then replace the former test case. After one iteration, the best and worst test cases will be re-selected, and the modification is done based on the new best and worst test cases. The process is iterated until the maximum number of generations is achieved. The best test case generated at the end of the iterative process is added into a temporary test suite. The whole process keeps repeating until all combinations of input values are fully covered as illustrated in Fig.  1.
Assume that the best candidate solution (best) has the best value of f(x) while the worst candidate solution (worst) has the worst value of f(x) in the candidate suite, and Xj,k,i is the value under the j th design variable of the kth candidate solution for the i th iteration. Thus, Eq. (1) is fulfilled the requirements.

Experimental Results and Discussion:
This section is divided into three parts which are parameter tuning of CTJ, two experiments for IOR and one experiment for uniform strength interaction strength to evaluate the difference between CTJ with another existing strategy in handling combinatorial optimization.

Parameter Tuning of CTJ
All experiments conducted are using Intel i7-6500U as the CPU with the RAM of 8GB in Windows 10 Professional operating system. There are only two common controlling parameters involved in CTJ which are population size and number of iterations. Hence, the tuning of parameters setting is performed to ensure the optimal results and efficiency of CTJ before the experiments are carried out. The tuning process is executed using one of the configurations from each IOR and uniform interaction strength experiments in (28) and (22). Three types of parameter settings stated in Table 1 are being experimented for 10 iterations to figure out which parameter setting will generate the most optimum and efficient result. The first experiment is adapted from (28) by using 10 parameters with 3 values each and the first 30th input-output relationships are utilized. From the result of the execution in Table 2, S1 setting is generated test cases in the shortest time but it yielded the highest number of test cases generated. S3 has the best number of generated test cases but it consumed a very long time to finish the execution. If compared to S2, S3 took approximately four times of S2 time to reduce five test cases to be generated in the best result. It is impractical to consume such a long time to reduce a small number of test cases. The number of test cases produced in S1 is reduced significantly compared to S2 which reduced 18 test cases. This result is much more optimum and acceptable to be used.  The second configuration of experiment is originated from (22) and the configuration is 6 parameters with 6 values each with uniform interaction strength of 3. The best and average number of test cases generated as well as the average execution time are stated in Table 3. S1 in this experiment is still the fastest parameter setting that completed the test case generation. However, the number of test cases it produced is still undesired compared to S2 and S3 settings. The time taken for the S3 setting to complete the generation of test cases is approximately 3 hours while S2 only took 55 minutes. The difference in the number of test cases generated between S3 and S2 is just eight test cases. These issues show that it is unrealistic to use an S3 setting in the real environment.
Based on both experiments that are conducted to decide the parameter settings, the S2 setting is selected to be parameter setting for all experiments since it is capable to generate the optimum number of test cases in a satisfactory time frame.

Experiments for Input-Output Based Relation
Two experiments published in (28) are conducted to evaluate the performance of CTJ in IOR. The strategies that have implemented the IOR feature which included in both experiments are Density (13), TVG (29), ReqOrder (30), ParaOrder (13), Union (31), Greedy (12), ITTDG (11) and AURA (16). Both experiments share the same input-output relationships, but their configuration of parameters and their values are different. The first experiment used IOR (N, 3 10 , R) configuration while the second experiment employed IOR (N, 2 3 , 3 3 , 4 3 , 5 1 , R) configuration. All IOR relationships are stated in Table 4. The experiment will be carried out for 6 IOR configurations. Initially, the first 10 relationships will be tested as the first IOR configuration and the second 10 relationships will be added into the first IOR configuration to become the second IOR configuration for the second evaluation. For the subsequence evaluations, another 10 relationships will be added into the previous configuration until all 60 relationships are assessed. The result of the first experiment which used IOR (N, 3 10 , R) configuration with the relationships in Table 4 is shown in Table 5  are taken by CTJ to complete the execution of R20, R30, R40, R50 and R60 respectively.  Table 6 is the result of the execution of experiment two in IOR. The most minimum number of test cases produced by CTJ in R10 and R20 only vary for 5 test cases if compared to the best algorithms. While R30 to R60, the difference between the best result of CTJ and Density is not more than 12 test cases. Additionally, the time taken to complete the execution in experiment two is in the range of 500 to 1000 seconds which approximately 8 to 16 minutes only. These show CTJ generates solutions that are close to optimum. The number of test cases generated through CTJ is still acceptable in overall if compared to the size of the test suite produced through exhaustive testing. Furthermore, the results produced by other strategies often go for a very high population size and a more iteration for improvement of the solutions while the parameters setting of CTJ for these experiments are only 500 iterations with the population size of 50. Different parameters setting will affect how well the strategy performed and hence resulting in a different size of test suite produced.

Conclusion:
In this paper, the existing strategies that are used in solving combinatorial testing such as Greedy, Density, AURA, ACO, GA, HS, PSO and SA have been studied. CTJ which stands for the combinatorial testing strategy that featured inputoutput based relation using Jaya algorithm is proposed and introduced. Referring to the results of the experiments, CTJ performs with acceptable performance especially in test case generation through input-output based relation. Overall, comparing the CTJ results to the results of existing test case generation strategies, CTJ generates more test cases compared to the other strategies, but this study focuses on generating test cases with acceptable test list size within short processing time. One of the constraints or challenges that face the testers is the test list generation time. The test list generation processing time needs to be as fast as possible to conduct the testing with a short time. So, in this research the test list generation processing time is the major factor. The search engine in CTJ is the main part which is based on the Jaya algorithm. Jaya algorithm depends on the global search approach to find the best solutions which may lead to moving far from the available solutions in the population space. Based on the low performance of the CTJ in the test case generation, the Jaya algorithm in CTJ must go through some modifications in order to improve the performance of CTJ and even generate a more optimum number of test cases. In addition, CTJ can be enhanced by adding the support of variable interaction strength, constraints, and seeding.