Overview
Systems in general, including software and hardware, usually have some input parameters that will affect the output produced by the system. The number of parameters and the possible values, their ranges, vary and can be finite or even infinite.
Taking a simple flight booking site as an example, we can easily have thousands of combinations for Departure Airport, Destination Airport, Class, Number of Adults, Number of Children input parameters.
This leads to a potential high-number of scenarios to be tested.
Testing
How can we test these systems?
At first sight, if we aim to test this system in-depth, we would need to test the system with all the possible combinations of values of these parameters.
This would:
- take long time
- be costly
- be eventually innefficient (more info ahead)
The first strategy would be on adopting data-driven testing.
Data-driven testing is a technique where a well-defined test script is executed multiple times, taking into account a "table" of parameters and corresponding values.
Usually, data-driven testing is used as a way to inject data to test automation scripts but it can also be used for manually performing the same test multiple times against different data.
The exact combination of parameter values to be used is beyond of the scope of data-driven testing.
Learn more
Xray has built-in support for datasets where testers can explicitly enumerate parameters and the combination of values to be tested. Xray also supports combinatorial parameters, where the user defines the values for each parameter and Xray calculates all the possible combinations, turning that into the dataset to be used.
Please see Parameterized Tests for more info.
Combinatorial testing is a "black-box test technique in which test cases are designed to exercise specific combinations of values of several parameters" (ISTQB).
Empyrical data
Several studies indicate that the vast majority of defects related to input values are due to either to a problem in a parameter or in a combination of two parameter values.