Page History
...
Systems in general, including software and hardware, usually have some input parameters that will affect the output(s) produced by the system. The number of parameters and the possible values, their ranges, vary and can be limited, huge but finite, or even infinite.
...
This leads to a potential high-number of scenarios to be tested. For the previous example, and considering a model where parameters have just a limited number of possible values, that would still lead to 3*3*3*2*3=162 scenarios.
| Info |
|---|
System produces A system generates different results/outputs not only due to changes on the input parameters but also due to using the system on different contexts (e.g., configurations, operating system, timezones, cloud provider). |
...
- takes considerable time
- is costly interms of human resources or infrastructure
- may be innefficient (more info ahead)
...
- )
...
| Info | ||
|---|---|---|
| ||
Xray also supports combinatorial parameters, where the user defines the values for each parameter and Xray calculates all the possible combinations, turning that into the dataset to be used.
It's possible to remove some values of the combinations to be generated. For example, we can exclude the "First" Class. That would lead to less scenarios to test (e.g., 162 => 108) but could still not be enough if we aim to have a limited set of tests.
Please see Parameterized Tests for more info. |
...
Studies, such as presented by NIST, PRACTICAL COMBINATORIAL TESTING, 2010, indicate that the vast majority of defects (67%-93%) related to input values are due to either to a problem in a parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).
NIST research showed that most software bugs and failures are caused by one or two parameters, with progressively fewer by three or more, which means that combinatorial testing can provide more efficient fault detection than conventional methods. Multiple studies have shown fault detection equal to exhaustive testing with a 20X to 700X reduction in test set size. New algorithms compressing combinations into a small number of tests have made this method practical for industrial use, providing better testing at lower cost. - NIST
The following chart (see reference ahead) represents the cumulative error detection rate for fault-triggering conditions.
The key insight underlying t-way combinatorial testing is that not every parameter contributes to every fault and many faults are caused by interactions between a relatively small number of parameters - Combinatorial Software Testing, Rick Kuhn and Raghu Kacker, National Institute of Standards and Technology, Yu Lei, University of Texas at Arlington, Justin Hunter, Hexawise
Single-value faults are mostly probable to typical mistakes, such as the off-by-one bug (e.g., imagine using a loop and using the "<" operator instead of "<="). The interaction of 2 parameters may be to bugs around implementing cascade conditional logic statements (e.g. using if or similar) involving those parameters/variables.
Bugs related to the interaction of more parameters decrease with the number of parameters; in other words, finding these rare bugs will require much more tests to be performed, leading to more time/costs. However, those rare t-way interaction faults can also be critical.
Combinatorial Testing considering 2-way (pairwise) and t-way interaction of parameters
Combinatorial testing is at first sight a technique that seems adequate precisely for this purpose, as the name shows.
| Info |
|---|
Combinatorial testing is a "black-box test technique in which test cases are designed to exercise specific combinations of values of several parameters" (ISTQB). |
How do we come with the exact combination of parameter values?
We could generate all possible combinations as mentioned earlier. However, and given Given the empyrical data mentioned earlier, adopting 2-way (pairwise) algorithms to test all the combinations of pairs of parameters (sometimes also called as "all-pairs testing") is a technique that is not only feasible but also provides great results in terms of fault-detection with a limited set of tests.
| Info |
|---|
There are different algorithms for generating t-way interaction of parameters (e.g., pairs, triplets). Different tools/algorithms may generate more scenarios than others to achieve the same coverage (in terms of interaction of parameters), and take more or less time. |
Reducing the number of test scenarios
Imagining the previous example, instead of having 162 test scenarios to perform, we would need just could have just 13 using pairwise to generate them.
Sometimes, we may need to test more thoroughly some parameters together, and for those we may opt for choose 3-way interactions, for example, to ensure that we cover all the combinations of values of 3 relevant parameters. In this case we We may even have mixed-strength scenarios where combinations of certain parameters may be are tested more thoroughly than others.
Having a limited set of test generated, we can then execute them. However, usually algoritms generate these tests Test scenarios are usually generated in a order, so that coverage is greater with the first tests and lesser with the last testsones. This way, if we stop testing at a given moment, we can make sure that we track coverage and that we tested the most combinations possible; we can track exactly the coverage of combinations with the number of executed tests.
In sum, there is a balance between the number of tests we execute and the coverage of interactions between variables (i.e., "t-way coverage") we will obtain.
...
| title | Please note |
|---|
...
validate.
Optimizing further the test scenarios to be performed
...