Page History
...
Taking a simple flight booking site as an example, we can easily have thousands of combinations for Flying From, Flying to, Class, (number of) Adults, (number of) Children input parameters.
It's possible to remove some values of the combinations to be generated. For example, we can exclude the "First" Class. That would lead to less scenarios to test (e.g., 162 => 108).
This leads to a potential high-number of scenarios to be tested. For the previous example, and considering a model where parameters have just a limited number of possible values, that would still lead to 3*3*3*2*3=162 scenarios.
In general, we could think that if we aim to test systems like this in-depth, we would need to test the system with all the possible combinations of values of these parameters.
But does it IIs it feasible? Does it even make sense? Are any of those scenarios redundant, or in other words, is there any manageaable subset of scenarios that can be tested that can still help us find bugs?
...
Test some well-known values for parameters
The first strategy that we may come with would be on adopting data-driven testing.
...
The exact combination of parameter values to be used is beyond of the scope of data-driven testing. However, usually testers include parameter combinations that represent examples coming as a direct consequence of acceptance criteria or from well-known "happy paths".
| Info | ||
|---|---|---|
| ||
Xray has built-in support for datasets where testers can explicitly enumerate parameters and the combination of values to be tested.
Please see Parameterized Tests for more info. |
Test every parameter combination
At first sight, if we aim to test this system in-depth, we would need to test the system with all the possible combinations of values of these parameters.
This would:
Testing every possible combination of parameters is only viable if we have very few parameters with very few possible values for each one of them.
In general, testing all combinations:
- takes considerable time
- is costly from human resources or infrastructure perspective
- may be
- take long time
- be costly
- be eventually innefficient (more info ahead)
...
| Info | ||
|---|---|---|
| ||
Xray also supports combinatorial parameters, where the user defines the values for each parameter and Xray calculates all the possible combinations, turning that into the dataset to be used.
It's possible to remove some values of the combinations to be generated. For example, we can exclude the "First" Class. That would lead to less scenarios to test (e.g., 162 => 108) but could still not be enough if we aim to have a limited set of tests.
Please see Parameterized Tests for more info.
|
Test using random combination of parameter values
Random testing is always an option that comes but it doesn't ensure we test combinations that matter unless we perform a very high number of tests, which would probabilisticly include a certain % of combinations or even all of them if we spend an infinite time randomly testing.
..Nobody wants to perform testing endlessly, without any sort of criteria. Random testing doesn't ensure we cover combinations that matter with a very limited set of tests.
Empyrical data
Several studies indicate that the vast majority of defects (67%-84%) related to input values are due to either to a problem in a parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).
...