Search

Testing is the process of evaluating a system by manual or automated means. While Regression Test Selection (RTS) discards test cases and Test Suite Minimization (TSM) shows diminution in fault detection rate, Test Case Prioritization (TCP) does not discard test cases. Test Case Prioritization techniques can be coverage or historical information based or model based. It can also be cost-time aware or requirement-risk aware. GUI/Web applications need special prioritization mechanism. In this paper, 90 scholarly articles ranging from 2001 to 2018 have been reviewed. We have explored IEEE, Wiley, ACM Library, Springer, Taylor & Francis and Elsevier database. We have also described each prioritization method with their findings and subject programs. This paper includes a chronological catalogue listing of the reviewed papers. We have framed three research questions which sum up the frequently used prioritization metrics, regularly used subject programs and the distribution of different prioritization techniques. To the best of our knowledge, this is the first review with a detail report of the last 18 years of TCP techniques. We hope this article will be beneficial for both beginners and seasoned professionals.

Regression testing is an expensive activity and Test Case Prioritization (TCP) acts as an improvement mechanism for it. TCP techniques for object oriented programs need attention and in our study, we explored prioritization of JUnit test cases. Ten benchmark Java programs with their several mutated versions were studied. As collecting coverage information is a costly effort, we bypassed these steps and used optimization heuristics for ordering JUnit test cases at test method level. Our approach formulated a novel fitness objective which depends on the number of modified lines executed per unit of execution time. As regression testing is performed after some modification is done on an existing program, maximizing the execution of number of modified lines is highly lucrative. The test case prioritization problem was replicated in context of 0/1 Knapsack problem and then it was solved using Genetic Algorithm (GA). Our exploration also included application of Simulated Annealing and Ant Colony Optimization method for determining the best execution ordering of test cases. We examined the usage of Multi-objective GA by building another new fitness metric which aims to maximize the number of inheritance edges covered by a test case. Results indicate the superiority of optimization heuristics over other existing approaches. It appeared that multi-objective GA yielded better result than single objective prioritization. Among the single objective techniques, ACO performed best. To the best of our knowledge, this is the first study which explored all the above mentioned optimization heuristics for ordering JUnit test cases with the newly coined fitness intents.

Testing ensures the delivery of a product in stipulated amount of time. According to Test Case Prioritization (TCP), test cases with higher priority get executed earlier. Enhancing the fault detection rate, capturing high priority requirements, reducing the cost and time of prioritization mechanism are several goals of TCP techniques. In this paper, we have proposed a fuzzy model that quantifies the savings generated by the prioritization mechanism. The model considers the effect of fault detection rate and number of test cases. Our first research question concentrates on implementing this fuzzy model while our second research question explores which technique for prioritization generates how much savings. Analysis of 10 C programs and 6 Siemens programs from Software Infrastructure Repository (SIR) shows that the technique of combining branch coverage and modified condition and decision coverage (MC/DC) yielded 2.01 times savings over non-prioritized test suite. Results also indicated that only branch coverage based TCP yielded 1.26 times savings over non-prioritized test suite. We hope our cost aware fuzzy model and these savings analysis will make test case prioritization more applicable with better planning. As several stages of software development do not have exact calculation of parameters, we realized that building a fuzzy model is of utmost importance.