Page History
...
| Info |
|---|
A system generates different results/outputs not only due to changes on the input parameters but also due to using the system on in different contexts (e.g., configurations, operating system, timezonestime zones, cloud provider). |
In general, we could think that if we aim to test systems like this in-depth, we would need to test the system with all the possible combinations of values of these parameters.
But does it even make sense? Are any of those scenarios redundant, or in other words, is there any manageaable manageable subset of scenarios that can be tested that can still help us find bugs?
...
The exact combination of parameter values to be used is beyond of the beyond the scope of data-driven testing. However, usually testers include parameter combinations that represent examples coming as a direct consequence of acceptance criteria or from well-known "happy paths".
...
- takes considerable time
- is costly interms in terms of human resources or infrastructure
- may be innefficient inefficient (more info ahead)
| Info | ||
|---|---|---|
| ||
Xray also supports combinatorial parameters, where the user defines the values for each parameter and Xray calculates all the possible combinations, turning that into the dataset to be used.
It's possible to remove some values of the combinations to be generated. For example, we can exclude the "First" Class. That would lead to less scenarios to test (e.g., 162 => 108) but could still not be enough if we aim to have a limited set of tests.
Please see Parameterized Tests for more info. |
...
Random testing is always an option that comes but it doesn't ensure we test combinations that matter unless we perform a very high number of tests, which would probabilisticly include probabilistically include a certain % of combinations or even all of them if we spend an infinite time randomly testing.
Nobody wants to perform testing endlessly, without any sort of criteria. Random testing doesn't ensure we cover combinations that matter with a very limited set of tests.
...
Empirical data about fault detection
Studies, such as presented by NIST, PRACTICAL COMBINATORIAL TESTING, 2010, indicate that the vast majority of defects (67%-93%) related to input values are due to either to a problem in a parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).
...
Single-value faults are mostly probable to typical mistakes, such as the off-by-one bug (e.g., imagine using a loop and using the "<" operator instead of "<="). The interaction of 2 parameters may be be due to bugs around implementing cascade conditional logic statements (e.g. using if or similar) involving those parameters/variables.
...
We could generate all possible combinations as mentioned earlier. However, and given the empyrical empirical data mentioned earlier, adopting 2-way (pairwise) algorithms to test all the combinations of pairs of parameters (sometimes also called as called "all-pairs testing") is a technique that provides great results in terms of fault-detection with a limited set of tests.
...
For example, in our flight booking scenario the Departure and Destination parameters need to be different. Also, we may have some rules in place where, for example, the First class is not available to children.
These are restrictons that restrictions that we can use in order to limit the generation of parameter combinations used by our test scenario.
...
| Info | ||
|---|---|---|
| ||
In Test Case Designer we can apply "constraints" involving the combination of 2 parameter values. We can apply several constrains constraints as shown in the followin following example: Class=First cannot exist together neither with Children=1 nor Children=Mode than 1.
|
...
The second level of optimization is about including important scenarios first.
Not all combination combinations of parameters may be equaly equally representative. Sometimes there are parameter combinations we know that are highly important as they represent highly used happy paths, or whose business impact is high.
...
| Info | ||
|---|---|---|
| ||
In Test Case Designer we can . In the following example, we considered an interaction that we need to test due to an hypothetic legislation a hypothetical legislation where some warning must be shown to users who are departing from the USA, using the First class, and have more than 1 childrenchild. That scenario will be added on the generated ones.
|
...
Pairwise and t-way testing don't tell us how to actually perform testing; it just generates the combination of parameters. Therefore, we can use this technique also if we choose to adopt a more exploratory testing approach, for example for certain configurations of hardware/software.
...
Test Case Designer (TCD) provides a more comprehensive modeling tool, where it's possible to define parameters, apply constraints, enforce specific scenarios, generate optimized datasets/scenarios using 2-way or t-way to ensure that certain parameters are tested together, up to a certain level. With Test Case Designer it's possible to have a limited and manageabled set of test scenarios to perform and make sure that most combination combinations of parameters are met with the initial scenarios, so that most risk is addressed upfront.
...