Page History
...
| Info | ||
|---|---|---|
| ||
Xray also supports combinatorial parameters, where the user defines the values for each parameter and Xray calculates all the possible combinations, turning that into the dataset to be used.
It's possible to remove some values of the combinations to be generated. For example, we can exclude the "First" Class. That would lead to less scenarios to test (e.g., 162 => 108) but could still not be enough if we aim to have a limited set of tests.
Please see Parameterized Tests for more info. |
...
What empirical studies tell us about fault detection
Studies, such as presented by NIST, PRACTICAL COMBINATORIAL TESTING, 2010, Before we talk about the testing option we consider optimal, let’s look at the underlying practical evidence.
Studies such as “PRACTICAL COMBINATORIAL TESTING, 2010”, presented by NIST, indicate that the vast majority of defects (67%-93%) related to input values are due to either to a problem in a either one parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).
...
The key insight underlying t-way combinatorial testing is that not every parameter contributes to every fault and many faults are caused by interactions between a relatively small number of parameters - Combinatorial Software Testing, Rick Kuhn and Raghu Kacker, National Institute of Standards and Technology, Yu Lei, University of Texas at Arlington, Justin Hunter, Hexawise
Single-value faults are mostly probable due to typical mistakes, such as the off-by-one bug (e.g., imagine using a loop and using the "<" operator instead of "<="). The interaction of 2 parameters may be due be due to bugs around implementing cascade conditional logic statements (e.g. using using if or similar) involving those parameters/ variables.
Bugs related to the interaction of more parameters decrease with the number of parameters; in other words, finding these rare bugs will require much more tests to be performed, leading to more time/costs. However, those rare t-way interaction faults can also be critical and address by proper testing techniques.
Combinatorial Testing considering 2-way (pairwise) and t-way interaction of parameters
...