Search

Modern software systems often consist of many different components, each with a number of options. Although unit tests may reveal faulty options for individual components, functionally correct components may interact in unforeseen ways to cause a fault. Covering arrays are used to test for interactions among components systematically. A two‐stage framework, providing a number of concrete algorithms, is developed for the efficient construction of covering arrays. In the first stage, a time and memory efficient randomized algorithm covers most of the interactions. In the second stage, a more sophisticated search covers the remainder in relatively few tests. In this way, the storage limitations of the sophisticated search algorithms are avoided; hence, the range of the number of components for which the algorithm can be applied is extended, without increasing the number of tests. Many of the framework instantiations can be tuned to optimize a memory‐quality trade‐off, so that fewer tests can be achieved using more memory. [ABSTRACT FROM AUTHOR]

In a complex component-based system, choices (levels) for components (factors) may interact to cause faults in the system behaviour. When faults may be caused by interactions among few factors at specific levels, covering arrays provide a combinatorial test suite for discovering the presence of faults. While well studied, covering arrays do not enable one to determine the specific levels of factors causing the faults; locating arrays ensure that the results from test suite execution suffice to determine the precise levels and factors causing faults, when the number of such causes is small. Constructions for locating arrays are at present limited to heuristic computational methods and quite specific direct constructions. In this paper three recursive constructions are developed for locating arrays to locate one pairwise interaction causing a fault. [ABSTRACT FROM AUTHOR]

Covering arrays find important application in software and hardware interaction testing. For practical applications it is useful to determine or b ound the minimum number of rows, CAN(t, k, v), in a covering array for given values of the parameters t, k, and v. Asymptotic upper bounds for CAN(t, k, v) have been established using the Stein--Lovász--Johnson strategy and the Lovasz local lemma. A series of improvements on these bounds is developed in this paper. First an estimate for the discrete Stein--Lovász--Johnson bound is derived. Then using alteration, the Stein--Lovasz--Johnson bound is improved upon, leading to a two-stage construction algorithm. Bounds from the Lovász local lemma are improved upon in a different manner, by examining group actions on the set of symbols. Two asymptotic upper bounds on CAN(t, k, v) are established that are tighter than the known bounds. A two-stage bound is derived that employs the Lovász local lemma and the conditional Lováasz local lemma distribution. [ABSTRACT FROM AUTHOR]

When κ factors each taking one of ν levels may affect the correctness or performance of a complex system, a test is selected by setting each factor to one of its levels and determining whether the system functions as expected (passes the test) or not (fails). In our setting, each test failure can be attributed to at least one faulty (factor, level) pair. A nonadaptive test suite is a selection of such tests to be executed in parallel. One goal is to minimize the number of tests in a test suite from which we can determine which (factor, level) pairs are faulty, if any. In this paper, we determine the number of tests needed to locate faults when exactly one (or at most one) pair is faulty. To do this, we address an equivalent problem, to determine how many set partitions of a set of size N exist in which each partition contains v classes and no two classes in the partitions are equal. [ABSTRACT FROM AUTHOR]

Numerous constructions of the best known covering arrays are effective only for specific numbers of symbols. Fusion replaces numerous symbols by one, and can thereby employ such constructions to produce useful covering arrays on fewer symbols. Augmentation instead replaces one symbol by many, permitting the construction of covering arrays from those with fewer symbols. Until this time, augmentation has been of limited value because it introduces substantial redundant coverage. Here a general augmentation method is improved upon by analyzing the classes of interactions to be covered and employing variants of covering arrays, quilting arrays, to reduce the redundancy introduced. For strengths four, five, and six, quilting arrays are produced that can be used in the refined augmentation to produce many best known covering arrays. [ABSTRACT FROM AUTHOR]

• IPO strategy generates Covering Perfect Hash Families efficiently. • Fast covering array generation for large scale combinatorial testing. • Many improvements to upper bounds on CAN. • CPHFs with subspace restrictions lead to many different small CAs. Combinatorial testing makes it possible to test large systems effectively while maintaining certain coverage guarantees. At the same time, the construction of optimized covering arrays (CAs) with a large number of columns is a challenging task. Heuristic and Metaheuristic approaches often become inefficient when applied to large instances, as the computation of the quality for new moves or solutions during the search becomes too slow. Recently, the generation of covering perfect hash families (CPHFs) has led to vast improvements to the state of the art for many different instances of covering arrays. CPHFs can be considered a compact form of a specific family of covering arrays. Their compact representation makes it possible to apply heuristic methods for instances with a much larger number of columns. In this work, we adapt the ideas of the well-known In-Parameter-Order (IPO) strategy for covering array generation to efficiently construct CPHFs, and therefore implicitly covering arrays. We design a way to realize the concept of vertical extension steps in the context of CPHFs and discuss how a horizontal extension can be implemented in an efficient manner. Further, we develop a horizontal extension strategy for CPHFs with subspace restrictions that identifies candidate columns greedily based on conditional expectation. Then using a local optimization strategy, a candidate may be adjoined to the solution or may replace one of the existing columns. An extensive set of computational results yields many significant improvements on the sizes of the smallest known covering arrays. [ABSTRACT FROM AUTHOR]

Sequential processes can encounter faults as a result of improper ordering of subsets of the events. In order to reveal faults caused by the relative ordering of t or fewer of v events, for some fixed t, a test suite must provide tests so that every ordering of every set of t or fewer events is exercised. Such a test suite is equivalent to a sequence covering array, a set of permutations on v events for which every subsequence of t or fewer events arises in at least one of the permutations. Equivalently it is a (different) set of permutations, a completely t-scrambling set of permutations, in which the images of every set of t chosen events include each of the t! possible "patterns." In event sequence testing, minimizing the number of permutations used is the principal objective. By developing a connection with covering arrays, lower bounds on this minimum in terms of the minimum number of rows in covering arrays are obtained. An existing bound on the largest v for which the minimum can equal t! is improved. A conditional expectation algorithm is developed to generate sequence covering arrays whose number of permutations never exceeds a specified logarithmic function of v when t is fixed, and this method is shown to operate in polynomial time. A recursive product construction is established when t = 3 to construct sequence covering arrays on vw events from ones on v and w events. Finally computational results are given for t ∈{3, 4, 5} to demonstrate the utility of the conditional expectation algorithm and the product construction. [ABSTRACT FROM AUTHOR]

Abstract: A cover starter is a single vector that yields a covering array under the actions of groups on both the columns and the symbols of the starter. The existence of this compact representation of covering arrays facilitates effective exhaustive and heuristic search. When the group action on symbols fixes a small number of symbols, such cover starters lead to covering arrays that embed covering arrays on fewer symbols. Lower bounds on the length of cover starters over specified groups are established, and extensive computational results are developed to improve upper bounds for numerous covering array numbers. [Copyright &y& Elsevier]

Abstract: Covering arrays with mixed alphabet sizes, or mixed covering arrays, are useful generalizations of covering arrays that are motivated by software and network testing. Suppose that there are k factors, and that the ith factor takes values or levels from a set G i of size g i . A run is an assignment of an admissible level to each factor. A mixed covering array, MCA , is a collection of N runs such that for any t distinct factors, , every t-tuple from occurs in factors in at least one of the N runs. When , an MCA is a CA . The mixed covering array number, denoted by MCAN , is the minimum N for which an MCA exists. In this paper, we focus on the constructions of mixed covering arrays of strength three. The numbers MCAN are determined for all cases with and for most cases with . [Copyright &y& Elsevier]

Abstract: A covering array is an array, in which in every subarray, each of the possible t-tuples over symbols occurs at least once. The parameter t is the strength of the array. Covering arrays have a wide range of applications for experimental screening designs, particularly for software interaction testing. A compact representation of certain covering arrays employs “permutation vectors” to encode subarrays of the covering array so that a covering perfect hash family whose entries correspond to permutation vectors yields a covering array. We introduce a method for effective search for covering arrays of this type using tabu search. Using this technique, improved covering arrays of strength 3, 4 and 5 have been found, as well as the first arrays of strength 6 and 7 found by computational search. [Copyright &y& Elsevier]

Abstract: A covering array is an array such that every sub-array contains all t-tuples from symbols at least once, where t is the strength of the array. One application of these objects is to generate software test suites to cover all t-sets of component interactions. Methods for construction of covering arrays for software testing have focused on two main areas. The first is finding new algebraic and combinatorial constructions that produce smaller covering arrays. The second is refining computational search algorithms to find smaller covering arrays more quickly. In this paper, we examine some new cut-and-paste techniques for strength three covering arrays that combine recursive combinatorial constructions with computational search; when simulated annealing is the base method, this is augmented annealing. This method leverages the computational efficiency and optimality of size obtained through combinatorial constructions while benefiting from the generality of a heuristic search. We present a few examples of specific constructions and provide new bounds for some strength three covering arrays. [Copyright &y& Elsevier]

Abstract: A covering array is an array such that every sub-array contains all t-tuples from symbols at least once, where t is the strength of the array. Covering arrays are used in experiments to screen for interactions among t-subsets of k components. Strength two covering arrays have been studied from numerous viewpoints, resulting in a variety of computational, direct, and recursive constructions. Consequently, it can be difficult to determine the smallest covering array that results from known construction. To address this, existence tables for the best currently known covering arrays are presented. In the process, a new direct construction from orthogonal arrays is also introduced. [Copyright &y& Elsevier]

There are many published algorithms for generating interaction test suites for software testing, exemplified by AETG, IPO, TCG, TConfig, simulated annealing and other heuristic search, and combinatorial design techniques. Among these, greedy one-test-at-a-time methods (such as AETG and TCG) have proven to be a reasonable compromise between the needs for small test suites, fast test-suite generation, and flexibility to accommodate a variety of testing scenarios. However, such methods suffer from the lack of a worst-case logarithmic guarantee on test suite size, while methods that provide such a guarantee at present are less efficient or flexible, or do not produce test suites that are competitive in size for practical testing scenarios. In this paper, a new algorithm establishes that efficient, greedy, one-test-at-a-time methods can indeed produce a logarithmic worst-case guarantee on the test suite size. In addition, this can be done while still producing test suites that are of competitive size, and in a time that is comparable to the published methods. It is deterministic, guaranteeing reproducibility. It generates only one candidate test at a time, permits users to ‘seed’ the test suite with specified tests, and allows users to specify constraints of combinations that should be avoided. Further, statistical analysis examines the impact of five variables used to tune this density algorithm for execution time and test suite size: weighting of density for factors, scaling of density, tie-breaking, use of multiple candidates, and multiple repetitions using randomization. Copyright © 2007 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]