Overview

Systems in general, including software and hardware, usually have some input parameters that will affect the output produced by the system. The number of parameters and the possible values, their ranges, vary and can be limited, huge but finite, or even infinite.

Taking a simple flight booking site as an example, we can easily have thousands of combinations for Departure Airport, Destination Airport, Class, Number of Adults, Number of Children input parameters.

This leads to a potential high-number of scenarios to be tested. Is it feasible? Does it even make sense? Are any of those scenarios redundant, or in other words, is there any manageaable subset of scenarios that can be tested that can still help us find bugs?

Testing

How can we test these systems?

At first sight, if we aim to test this system in-depth, we would need to test the system with all the possible combinations of values of these parameters.

...

Combinatorial testing is a "black-box test technique in which test cases are designed to exercise specific combinations of values of several parameters" (ISTQB). 

Empyrical data

Several studies indicate that the vast majority of defects (67%-84%) related to input values are due to either to a problem in a parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).

Bugs related to the interaction of more parameters decrease with the number of parameters; in other words, find these rare bugs will require much more tests to be performed, leading to more time/costs.

Testing

Given the empyrical data, adopting pairwise testing to test all the ombinations of pairs of parameters (sometimes also called as "all pairs testing") is a technique that is feasible.

Imagining the previous example, instead of having XXX test scenarios to perform, we would need just XX.

Sometimes, we may need to test more thoroughly some parameters, and for those we may choose to 3-way testing, for example, to ensure that we cover all the combinations of values of 3 relevant parameters.

Having a limited set of test generated, we can then execute them. However, usually algoritms generate these tests in a order, so that coverage is greater with the first tests and lesser with the last tests. This way, if we stop testing at a given moment, we can make sure that we track coverage and that we tested the most combinations possible.

In sum, there is a balance between the number of tests we execute and the coverage we will obtain.