Page History
...
This leads to a potential high-number of scenarios to be tested. Is it feasible? Does it even make sense? Are any of those scenarios redundant, or in other words, is there any manageaable subset of scenarios that can be tested that can still help us find bugs?
Testing
At first sight, if we aim to test this system in-depth, we would need to test the system with all the possible combinations of values of these parameters.
This would:
- take long time
- be costly
- be eventually innefficient (more info ahead)
In this tutorial we'll learn about the testing challenges of these systems and how to overcome them efficiently.
Initial testing options
Test some well-known values for parameters
The first strategy would be on adopting data-driven testing.
...
| Info | ||
|---|---|---|
| ||
Xray has built-in support for datasets where testers can explicitly enumerate parameters and the combination of values to be tested. Xray also supports combinatorial parameters, where the user defines the values for each parameter and Xray calculates all the possible combinations, turning that into the dataset to be used. Please see Parameterized Tests for more info. |
Test every parameter combination
At first sight, if we aim to test this system in-depth, we would need to test the system with all the possible combinations of values of these parameters.
This would:
- take long time
- be costly
- be eventually innefficient (more info ahead)
Combinatorial testing is a "black-box test technique in which test cases are designed to exercise specific combinations of values of several parameters" (ISTQB).
Test using random combination of parameter values
Random testing doesn't ensure we test combinations that matter.
...
Empyrical data
Several studies indicate that the vast majority of defects (67%-84%) related to input values are due to either to a problem in a parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).
Bugs related to the interaction of more parameters decrease with the number of parameters; in other words, find these rare bugs will require much more tests to be performed, leading to more time/costs.
Testing
Given the empyrical data, adopting pairwise testing to test all the ombinations of pairs of parameters (sometimes also called as "all pairs testing") is a technique that is feasible.
...