DEO: A Dynamic Event Order Strategy for t-way Sequence Covering Array Test Data Generation

Sequence covering array (SCA) generation is an active research area in recent years. Unlike the sequence-less covering arrays (CA), the order of sequence varies in the test case generation process. This paper reviews the state-of-the-art of the SCA strategies, earlier works reported that finding a minimal size of a test suite is considered as an NP-Hard problem. In addition, most of the existing strategies for SCA generation have a high order of complexity due to the generation of all combinatorial interactions by adopting one-test-at-a-time fashion. Reducing the complexity by adopting one-parameterat-a-time for SCA generation is a challenging process. In addition, this reduction facilitates the supporting for a higher strength of coverage. Motivated by such challenge, this paper proposes a novel SCA strategy called Dynamic Event Order (DEO), in which the test case generation is done using one-parameter-at-a-time fashion. The details of the DEO are presented with a step-by-step example to demonstrate the behavior and show the correctness of the proposed strategy. In addition, this paper makes a comparison with existing computational strategies. The practical results demonstrate that the proposed DEO strategy outperforms the existing strategies in term of minimal test size in most cases. Moreover, the significance of the DEO increases as the number of sequences increases and/ or the strength of coverage increases. Furthermore, the proposed DEO strategy succeeds to generate SCAs up to t=7. Finally, the DEO strategy succeeds to find new upper bounds for SCA. In fact, the proposed strategy can act as a research vehicle for variants future implementation.


Introduction:
T-way test case generation (also, termed combinatorial testing) strategies have been adopted as effective black-box testing strategies for many systems under test (SUT). In which, a subset of exhaustive test cases is generated to cover the interaction between parameters for certain strength (termed t-way strength) at least one. Combinatorial test case generation can be either sequence-less or sequence based process, the resulting test suites are called either Covering Array (CA) or Sequence (Event) Covering Array (SCA), respectively (1(. Earlier works in combinatorial testing have been reported that the generation of test cases is considered both NP-Complete (i.e., there is no unique solution for covering tuples) and NP-Hard problems (there is no unique solution that always generates minimal test size) (2,3).
Thus, many strategies exist in the literature. Some of these strategies are based on algebraic algorithms in which the test size is near minimal (2,3). Nevertheless, search based computational strategies are also considered as competitive strategies and can produce minimal test suite for some configurations (1,3).
The test suite generation of computational strategies involves two approaches: one-test-at-atime and one-parameter-at-a-time. In the first approach, the generation of all t-way interaction elements (tuples) is followed by a search algorithm that generates a test case that maximizes the covered tuples. The generated test case is added to the test suite. This process is repeated for each test case until all tuples are covered. This approach is adopted for both CAs and SCAs generation. In contrast, in one-parameter-at-a-time approach, the interaction elements are subdivided in the generation for each input parameter. The test suite starts from t parameters covered. Then through horizontal extension, a new parameter is considered to be added at the pre-generated test case. Next, a vertical expansion is followed to cover the remaining tuples like the one-test-at-a-time approach. Thus, one-parameter-at-a-time achieves a lower order of complexity than that of one-test-ata-time (3)(4)(5)(6). For this reason, this paper proposes a strategy based on one-parameter-at-a-time method. The remaining sections of this paper give a literature review on combinatorial test data generation strategies and focus on SCA notations and applications. Followed by the proposed strategy with a step-by-step example. Next, an evaluation, comparison with the existing works, and critical analysis of the results are discussed. The final section states the conclusion and future directions.
Deterministic means running the algorithm several times yield the same test suite. Although deterministic nature is more preferred for a tester, Non-deterministic some-times generate different test size; thus it may generate a minimal test suite (1,6,7).
SCAs unlike CAs in history; CAs research started in the last 20 years for t=2 (pairwise testing) then some algorithms are generalized for t-way testing in the last decade. Whilst, SCAs is first identified as a combinatorial explosion problem by National Institute of Standards and Technology (NIST) research group in 2012 (20). SCA (N, t, n) as an N x n matrix where entries are from a finite Sequence Set SS of n events, such that every t-length permutation of symbols from SS occurs in at least one row (21). The test size is N. To understand the effectiveness of SCA combinatorial test case generation, for a system with 10 events, the exhaustive testing generates 10! In 2012, Kuhn et al. presented a strategy called t-way Sequence (TSEQ). In TSEQ a greedy algorithm is adopted to facilitate the selection among candidate test cases to select the highest score test case that covers the maximum number of t-way sequence combination until all combinations are covered in a deterministic one-test-at-a-time fashion (21). In addition, Kuhn et al. proved that an optimal solution for t=2 is quite simple by considering the sequence in order and reverse the sequence as a second test case which always generates a lower bound SCA of size 2 (i.e., SCA (2, 2, n), where n ≥2).
In 2012, Zabil et al. proposed a nondeterministic strategy based on the population Bee Algorithm (BA) (22). In BA, the t-way tuples present the food and the bees search the best source in one-test-at-a-time fashion.
In 2013, Chee et al. presented two deterministic algorithms namely: upper bound algorithm (U) and its reversal (Ur) (2). The U algorithm is based on a greedy method that selects one permutation set from t-way sets covering the most uncovered tuples. The Ur algorithm like U algorithm but a reverse order of the generated test case is also appended at the end of the test set (2).
In 2014, Rahman et al. adopted the SA algorithm for generation non-deterministic SCA called EDIST-SA (23). However, EDIST-SA is a conceptual strategy that reported a lower bound SCA for t=3 and merely four input sequence to generate a SCA of size 6 (i.e., SCA (6,3,4)).
In 2016, Ahmad et al. proposed sequence covering array testing generator (SCAT). SCAT selects one of the best candidates from candidates' pool deterministically in greedy one-test-at-a-time fashion (24).
In 2017, Rabbi proposed a swarm intelligence sequence generator (SISEQ). The SISEQ selects the particles, which in turn, identify the search domain. Next, the SISEQ determines the global velocity to maximize the tuples covered in a greedy one-test-at-a-time manner (25). Later, Rabbi proposed a modification to SISEQ (mSISEQ) to tackle the complexity of the search space for t=4. In some results, mSISEQ outperforms the original SISEQ as generates minimal test size.
In 2018, Nasser et al. proposed an enhanced exploration capability via elitism (eFPA) (1) for its counterpart FPA. Unlike all existing works, both strategies can generate CAs and SCAs in the same implementation. Moreover, these non-deterministic strategies are reported SCAs up to t=6.

DEO Strategy:
DEO strategy adopts the one-parameter-ata-time approach. Unlike the one-parameter-at-atime family that takes the parameter-values in order for CAs generation, the proposed DEO search the order dynamically for both horizontal extension and vertical expansion. The DEO strategy is illustrated in Fig. 1. In order to demonstrate the proposed DEO strategy and demonstrate the correctness. an illustrative example is to be considered for sequence set= (0,1,2,3,4), t=3. The example uses the symbols defined in Fig. 1. The DEO starts with empty π and ts, t=3, n=5. The DEO denotes E1E2E3= [0,1,2], then generates 3! combinations, appends them to ts, as tabulated in Table 1. Next, i=4, and the Π set consists of 3! combinations of sequences E1E2E3= [1,2,3], [0,2,3] and [0, 1,3]. Here, it should be mentioned that there is no need to generate the previously generated sequences (i.e., for E1E2E3= [0,1,2]) since these combinations are already covered in previous steps. Thus, the Π set for 3-way combinations is listed in Fig. 2.  initialize ts as an empty set; 3.
let n be the size of ss; 4.
let E1, E2, …, and En be the events in ss in an arbitrary order; 5.
generate t! permutation (t-way combination of events) of the first t events; 6.
append the generated tuples in step 4 to ts; 7.
let π be t-way permutations of sequences set involving the event Ei and the t-1 events among the previous i-1 events; 10.
starts from the end to the first position; 14.
select a sequence position of Ei; 15.
choose the first tuple from π and do the exhaustive search for missing events such that the combined test case (τ) covers the maximum number of tuples in π set; 22.

Figure 3. The Π set for 3-way combinations when i=5.
The horizontal extension for the first six test cases is tabulated in The vertical expansion starts with tuple [1,4,3] and does an exhaustive search for this missing sequences (i.e., 0 and 2 in this case). This yields a test case "0, 2, 1, 4, 3" which covers two tuples (i.e., [1,4,3], and [2,1,4]). Next, this test case is appended to the test case list (i.e., test case number 7), and the tuples covered are deleted from the Π set. By the same way, the final test case "3, 4, 1, 0, 2" which covers the remaining two tuples are appended to the test case list (i.e., test case number 8), and the Π set is empty. Thus, during the vertical expansion, the test cases are expanded to cover the uncovered tuples in horizontal extension. The SCA (8, 3, 5) is tabulated in Table 3.
In order to tackle the significant of DEO as a minimization test case generation, we introduce the delta (∆) as the difference in size between the DEO's result (NDEO) and the best-known result (NBest) as given in Eq. 1. Thus, when ∆=0 this means that DEO generates the same upper bound of well-known results. When ∆ has a positive value, it means that there is a diverging from a minimal solution. Similarly, when ∆ has a negative value, it means that DEO contributes to finding a new upper bound for a minimal solution, the significant increased as the value increased. ∆ = NDEO -NBest Equation (1) Referring to Table 4 when t=3, BA, FPA, eFPA, and DEO starts with lower boundary when n=4; which is identical to the best solution of EDIST-SA. Both FPA and eFPA outperforms existing strategies when n=5. eFPA outperforms existing strategies when n=8. Nevertheless, DEO outperforms the other strategies for higher number of events and produces new upper bounds for SCA when n= 13,14,15,16,18,19, and 20. Referring to Table 5 when t=4, SCAT, SISEQ, and DEO start with lower bound size when n=5. BA, SCAT, TSEQ, FPA, and eFPA outperforms DEO when n=6. mSISEQ outperforms existing strategies when n=7and n=9. SISEQ, mSISEQ, and eFPA outperform existing strategies when n=8. Whilst, DEO outperforms the existing strategies when 10≤n≤20. It should be mentioned that the significant of DEO increases as the number of events increased. Referring to Table 6 when t=5, eFPA outperforms DEO when n=7. Whilst, DEO outperforms the existing strategies and generates new upper bound for SCA when n=6, 8, 9, and 10. Referring to Table 7 when t=6, DEO outperforms the existing strategies when 7≤n≤10 significantly and produces new upper bound for SCA. There is a little attention for providing higher degree of interaction for SCA in the literature (i.e., t>=7). For this reason, only DEO generates SCAs as tabulated in Table 8. Thus, ∆ is kept empty when t=7.

Conclusions:
This paper proposed a strategy called DEO based on one-event-at-a-time for t-way SCA test data generation. The proposed strategy outperforms most of the existing works as far as minimal test size is concerned. In addition, unlike the existing works, DEO can generate SCAs up to t=7.Moreover, the significant of DEO increases as the number of sequences (n) increases as well as the strength of coverage (t) increases. Furthermore, new upper bounds of SCAs are reported. Apart from future works, we are currently studying some variants for both horizontal extension and vertical expansion which may lead to further minimization to the test suite. Another direction for future work is to make a hybrid strategy based on pre-generated SCA and MVSCA.