Page History
...
Systems in general, including software and hardware, usually have some input parameters that will affect the output(s) produced by the system. The number of parameters and the possible values, their ranges, vary and can be limited, huge but finite, or even infinite.
A system generates different results/outputs not only due to changes in the obvious input parameters but also due to using the system in different contexts (e.g., configurations, operating system, time zones, cloud provider). Those types of variation ideas should also be accounted for in your testing.
Taking a simple flight booking site as an example, we can easily have thousands of combinations for Flying From, Flying to, Class, (number of) Adults, (number of) Children input parameters.
This potentially leads to a potential high - number of scenarios to be tested . For - for the previous example , and considering a model where parameters have just model above, with a limited number of possible values, that would still lead to 3*3*3*2*3=162 scenarios.
...
.
...
In general, we could think that if we aim to test systems like this in-depth, we would need to test the system them with all the possible combinations of values of these parameters.
But does it even make sense? Are any of those scenarios redundant, or in other words, is there any manageable subset of scenarios that can be tested that can still help us find bugs?
In this tutorial we'll learn about the testing challenges of these systems and how to overcome them efficiently.
| Info |
|---|
A system generates different results/outputs not only due to changes on the input parameters but also due to using the system in different contexts (e.g., configurations, operating system, time zones, cloud provider). |
Initial testing options
Option 1: Test using
...
“familiar” selection of values for the parameters
The first strategy that we may come up with would be adopting is data-driven testing. Data-driven testing It is a technique where a well-defined test script is executed multiple times, taking into account a "table" of parameters and corresponding values.
Usually, data-driven testing is used as a way to inject data to test automation scripts but it can also be used for to manually performing perform the same test multiple times against different data iterations.
The However, the exact combination of parameter values to be used is beyond the beyond the scope of data-driven testing. However Usually, usually testers include parameter value combinations that represent examples coming as a direct consequence of acceptance criteria or , from well-known "happy paths", or from the production data.
| Info | ||
|---|---|---|
| ||
Xray has built-in support for datasets where testers can explicitly enumerate parameters and the combination of values to be tested.
Please see Parameterized Tests for more info. |
Option 2: Test using random combination of parameter values
Random testing is always an option but it doesn't ensure we test combinations that matter unless we perform a very high number of tests, which would probabilistically include a certain % of combinations or even all of them if we spend an infinite time randomly testing.
Nobody wants to perform testing endlessly, without any sort of criteria. Random testing doesn't ensure we cover combinations that matter with a very manageable set of tests.
Option 3: Test every parameter/value combination
Testing every possible combination of parameters is only viable if we have very few parameters with very few possible values for each one of them.
In general, testing all combinations:
- takes considerable time
- is costly in terms of human resources or infrastructure
...
| Info | ||
|---|---|---|
| ||
Xray also supports combinatorial parameters, where the user defines the values for each parameter and Xray calculates all the possible combinations, turning that into the dataset to be used.
It's possible to remove some values of the combinations to be generated. For example, we can exclude the "First" Class. That would lead to less scenarios to test (e.g., 162 => 108) but could still not be enough if we aim to have a limited set of tests.
Please see Parameterized Tests for more info. |
Test using random combination of parameter values
...
. |
Nobody wants to perform testing endlessly, without any sort of criteria. Random testing doesn't ensure we cover combinations that matter with a very limited set of tests.
Empirical data about fault detection
...
Combinatorial Testing considering 2-way (pairwise) and t-way interaction of parameters
Combinatorial testing is at first sight a technique that seems adequate precisely for this purpose, as the name shows.
...
Given the empirical data mentioned earlier, adopting combinatorial testing is an approach that provides great results in terms of fault detection/defect slippage prevention with manageable test suite size.
It is a "black-box test technique in which test cases are designed to exercise specific combinations of values of several parameters" (ISTQB).
...
How do we come with the exact combination of parameter values?
We could generate all possible combinations as mentioned earlier. However, and given the empirical data mentioned earlier, adopting 2-way (pairwise) algorithms to test all the combinations of pairs of parameters (sometimes also called "all-pairs testing") is a technique that provides great results in terms of fault-detection with a limited set of tests.
...
Instead of letting a human select them by hand, we rely on tools to perform it more efficiently. There are different algorithms for generating t-way
...
interactions of parameters (e.g., pairs, triplets)
...
- some may create more scenarios than others to achieve the same coverage (in terms of interaction of parameters), and take more or less time. There are a couple of important algorithm features to consider as seen ahead.
Reducing the number of test scenarios and the time to create them
The core function of combinatorial algorithms is identifying the smallest mathematically-possible set of scenarios to satisfy the t-way condition, and doing that much faster than in the manual approach.
Imagining the previous example, instead of having 162 test scenarios to perform, we could have just have just 13 using the pairwise technique to generate them.
Sometimes, we may need to test more thoroughly some parameters together, and for those we may choose 3-way interactions, for example, to ensure that we cover, e.g., all the combinations of 3 values of 3 relevant parameters. across all parameters, which would still require only 35 scenarios.
We may even have mixed-strength scenarios risk-based use cases where combinations of certain parameters are tested more thoroughly than others.
Test scenarios are usually generated in a specific order, so that the interaction coverage is greater with the first tests and lesser with the last ones. This way, if we stop testing at a given moment, we can make sure that we tested the most combinations possible; we can some tools, such as Test Case Designer, allow us to track exactly the coverage of combinations achieved with the a given number of executed tests.
In sum, there is a balance between the number of tests we execute and the coverage of interactions between variables (i.e., "t-way coverage") we validate.
...
Combining business acumen with statistical techniques
The first level of optimization is further reducing the number of generated test scenario.
...