An exploratory study of history-based test case prioritization techniques on different datasets
Main Article Content
Abstract
In regression testing, Test case prioritization (TCP) is a technique to arrange all the available test cases. TCP techniques can improve fault detection performance which is measured by the average percentage of fault detection (APFD). History-based TCP is one of the TCP techniques that consider the history of past data to prioritize test cases. The issue of equal priority allocation to test cases is a common problem for most TCP techniques. However, this problem has not been explored in history-based TCP techniques. To solve this problem in regression testing, most of the researchers resort to random sorting of test cases. This study aims to investigate equal priority in history-based TCP techniques. The first objective is to implement different history-based TCP techniques. The second objective is to explore the problem of equal priority in history-based TCP techniques. The third objective is to explore random sorting as a solution to the problem of equal priority in history-based TCP techniques. Datasets of historical records of test cases from conventional and modern sources were collected. History-based TCP techniques were applied to different datasets. The History-based TCP techniques were checked for the problem of equal priority. Then random sorting was used as a solution to the problem of equal priority. Finally, the results were elaborated in terms of APFD and execution time. The results indicate that history-based techniques also suffer from the problem of equal priority like other types of TCP techniques. Secondly, random sorting does not produce optimal results while trying to solve the problem of equal priority in history-based TCP. Furthermore, random sorting deteriorates the results of history-based TCP techniques when employed to solve the problem of equal priority. One should resort to random sorting if no other solution exists. The decision to choose the best solution requires a cost-benefit analysis keeping in view the context and solution under consideration.
Received 22/09/2023
Revised 10/02/2024
Accepted 12/02/2024
Published 25/02/2024
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
How to Cite
References
Younis MI, Alsewari AR, Khang NY, Zamli KZ. CTJ: Input-output based relation combinatorial testing strategy using jaya algorithm. Baghdad Sci. J. . 2020 Sep 8;17(3 (Suppl.)):1002-1009. https://dx.doi.org/10.21123/bsj.2020.17.3(Suppl.).1002
Khatibsyarbini M, Isa MA, Jawawi DN, Tumeng R. Test case prioritization approaches in regression testing: A systematic literature review. Inf. Softw. Technol.. 2018 Jan 1; 93:74-93. https://doi.org/10.1016/j.infsof.2017.08.014
Gupta A, Mahapatra RP. Multifactor Algorithm for Test Case Selection and Ordering. Baghdad Sci. J.. 2021 Jun 20;18(2 (Suppl.)):1056-1075. http://dx.doi.org/10.21123/bsj.2021.18.2(Suppl.).1056
Bajaj A, Sangwan OP. A systematic literature review of test case prioritization using genetic algorithms. IEEE Access. 2019; 7:126355–75. https://doi.org/10.1109/access.2019.2938260
Hao D, Zhang L, Zhang L, Rothermel G, Mei H. A unified test case prioritization approach. ACM Trans. Softw. Eng. Methodol.. 2014;24(2):1–31. https://dx.doi.org/10.1145/2685614
Hao D, Zhang L, Zang L, Wang Y, Wu X, Xie T. To be optimal or not in test-case prioritization. IEEE Trans. Softw. Eng.. 2016;42(5):490–505. https://dx.doi.org/10.1109/tse.2015.2496939
Srikanth H, Hettiarachchi C, Do H. Requirements based test prioritization using risk factors: An industrial study. Inf. Softw. Technol.. 2016; 69:71–83. https://dx.doi.org/10.1016/j.infsof.2015.09.002
Lou Y, Chen J, Zhang L, Hao D. Chapter one-a survey on regression test-case prioritization. vol. 113 of Adv. Comput. 2019;Volume 113:Pages 1-46 https://doi.org/10.1016/bs.adcom.2018.10.001
Dalal S, Assistant Professor, Maharshi Dayanand University, Rohtak, India. Challenges of regression testing: A pragmatic perspective. Int. J. Adv. Res. Comput. Sci. 2018;9(1):499–503. https://dx.doi.org/10.26483/ijarcs.v9i1.5424
Ling X, Agrawal R, Menzies T. How different is test case prioritization for open and closed source projects? IEEE Trans. Softw Eng. 2022;48(7):2526–40. https://doi.org/10.1109/tse.2021.3063220
Kim JM, Porter A. A history-based test prioritization technique for regression testing in resource constrained environments. In Proc. - 24th Int. Conf. Softw. Eng.. 2002 May 19 (pp. 119-129). https://doi.org/10.1145/581339.581357
Marijan D, Gotlieb A, Sen S. Test case prioritization for continuous regression testing: An industrial case study. In: 2013 IEEE Int. Conf. softw. maint. .IEEE. 2013. https://doi.org/10.1109/icsm.2013.91
Cho Y, Kim J, Lee E. History-based test case prioritization for failure information. In 2016 Proc. - 23rd Asia-Pac. Softw. Eng. Conf. (APSEC). IEEE; 2016. https://doi.org/10.1109/apsec.2016.066
Zhu Y, Shihab E, Rigby PC. Test re-prioritization in continuous testing environments. In 2018 IEEE Int. Conf. Softw. Maint. and Evol. (ICSME) 2018 ;Sep 23 :pp. 69-79. IEEE. https://doi.org/10.1109/icsme.2018.00016
Fazlalizadeh Y, Khalilian A, Azgomi MA, Parsa S. Prioritizing test cases for resource constraint environments using historical test case performance data. In: 2009 2nd IEEE Int. Conf. Comput. Sci. Inf. Tech.. IEEE; 2009. https://doi.org/10.1109/ICCSIT.2009.5234968
Hemmati H, Fang Z, Mantyla MV. Prioritizing manual test cases in traditional and rapid release environments. In: 2015 IEEE 8th Int. Conf. Softw. Test., Verif. and Valid.. (ICST). IEEE. 2015. https://doi.org/10.1109/ICST.2015.7102602
Yu Z, Fahid F, Menzies T, Rothermel G, Patrick K, Cherian S. TERMINATOR: better automated UI test case prioritization. In 2019: Proc. - 27th ACM Joint Meeting on Eur. Softw. Eng. Conf. and Symp. on Found. Softw. Eng.. New York, NY, USA: ACM; 2019. https://doi.org/10.1145/3338906.3340448
Qasim M, Bibi A, Hussain SJ, Jhanjhi NZ, Humayun M, Sama NU. Test case prioritization techniques in software regression testing: An overview. Int. j. adv. appl. sci. .2021 May;8(5):107-21. https://doi.org/10.21833/ijaas.2021.05.012
Khatibsyarbini M, Isa MA, Jawawi DN, Hamed HN, Suffian MD. Test case prioritization using firefly algorithm for software testing. IEEE access. 2019 Sep 10; 7:132360-73. https://doi.org/10.1109/ACCESS.2019.2940620
Chen J, Gu Y, Cai S, Chen H, Chen J. A novel test case prioritization approach for black‐box testing based on K‐medoids clustering. J. Softw.: Evol. Process. 2023 Mar 31; e2565. https://doi.org/10.1002/smr.2565
Mukherjee R, Patnaik KS. A survey on different approaches for software test case prioritization. J. King Saud Univ. - Comput. Inf. Sci. .2021 Nov 1;33(9):1041-54. https://doi.org/10.1016/j.jksuci.2018.09.005
Zhou ZQ, Liu C, Chen TY, Tse TH, Susilo W. Beating random test case prioritization. IEEE Trans. Reliab. . 2020 Jun 16;70(2):654-75. https://doi.org/10.1109/TR.2020.2979815
Lu C, Zhong J, Xue Y, Feng L, Zhang J. Ant colony system with sorting-based local search for coverage-based test case prioritization. IEEE Trans. Reliab.. 2019 Aug 9;69(3):1004-20. https://doi.org/10.1109/TR.2019.2930358
Mahdieh M, Mirian-Hosseinabadi SH, Mahdieh M. Test case prioritization using test case diversification and fault-proneness estimations. Autom. Softw. Eng. .2022 Nov;29(2):50. https://doi.org/10.1007/s10515-022-00344-y
Ganjkhani E, Afsharchi M. An effective test case prioritization by combination of strategies. SN Appl. Sci. .2019 Sep; 1:1-4. https://doi.org/10.1007/s42452-019-1076-1
Chiang CL, Huang CY, Chiu CY, Chen KW, Lee CH. Analysis and assessment of weighted combinatorial criterion for test suite reduction. Qual. Reliab. Eng. Int.. 2022 Feb;38(1):358-88. https://doi.org/10.1002/qre.2984
Mondal S, Nasre R. Mahtab: Phase-wise acceleration of regression testing for C. J. Syst. Softw.. 2019 Dec 1; 158:110403. https://doi.org/10.1016/j.jss.2019.110403
Nithya TM, Chitra S. Soft computing-based semi-automated test case selection using gradient-based techniques. Soft Comput.. 2020 Sep;24(17):12981-7. https://doi.org/10.1007/s00500-020-04719-9
Jatana N, Suri B. Particle swarm and genetic algorithm applied to mutation testing for test data generation: a comparative evaluation. J. King Saud Univ. - Comput. Inf. Sci.. 2020 May 1;32(4):514-21. https://doi.org/10.1016/j.jksuci.2019.05.004
Qureshi N, Mukhija MK, Kumar S. RAFI: Parallel Dynamic Test-suite Reduction for Software. New Front. Commun. Intell. Syst, SCRS, India. 2021:165-76. https://doi.org/10.52458/978-81-95502-00-4-20
Wang R, Li Z, Jiang S, Tao C. Regression test case prioritization based on fixed size candidate set ART algorithm. J. Softw. Eng. Knowl. Eng.. 2020 Mar;30(03):291-320. https://doi.org/10.1142/s0218194020500138
Rothermel G, Untch RH, Chu C, Harrold MJ. Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng.. 2001 Oct;27(10):929-48. https://doi.org/10.1109/32.962562
Khatibsyarbini M, Isa MA, Jawawi DN, Shafie ML, Wan-Kadir WM, Hamed HN, Suffian MD. Trend application of machine learning in test case prioritization: A review on techniques. IEEE Access. 2021 Dec 14;9:166262-82. https://doi.org/10.1109/ACCESS.2021.3135508
Yan R, Chen Y, Gao H, Yan J. Test case prioritization with neuron valuation based pattern. Sci. Comput. Program.. 2022 Mar 1; 215:102761. https://doi.org/10.1016/j.scico.2021.102761
Huang Y, Shu T, Ding Z. A learn-to-rank method for model-based regression test case prioritization. IEEE Access. 2021 Jan 20; 9:16365-82. https://doi.org/10.1109/ACCESS.2021.3053163
Laaber C, Gall HC, Leitner P. Applying test case prioritization to software microbenchmarks. Empir. Softw. Eng.. 2021 Nov;26(6):133. https://doi.org/10.1007/s10664-021-10037-x