Overview

Systems in general, including software and hardware, usually have some input parameters that will affect the output produced by the system. The number of parameters and the possible values, their ranges, vary and can be limited, huge but finite, or even infinite.

Taking a simple flight booking site as an example, we can easily have thousands of combinations for Departure Airport, Destination Airport, Class, Number of Adults, Number of Children input parameters.

This leads to a potential high-number of scenarios to be tested. Is it feasible? Does it even make sense? Are any of those scenarios redundant, or in other words, is there any manageaable subset of scenarios that can be tested that can still help us find bugs?

In this tutorial we'll learn about the testing challenges of these systems and how to overcome them efficiently.

Initial testing options

Test some well-known values for parameters

The first strategy would be on adopting data-driven testing.

Data-driven testing is a technique where a well-defined test script is executed multiple times, taking into account a "table" of parameters and corresponding values.

Usually, data-driven testing is used as a way to inject data to test automation scripts but it can also be used for manually performing the same test multiple times against different data.

The exact combination of parameter values to be used is beyond of the scope of data-driven testing.

Test every parameter combination

At first sight, if we aim to test this system in-depth, we would need to test the system with all the possible combinations of values of these parameters.

This would:

1. take long time
2. be costly
3. be eventually innefficient (more info ahead)

Combinatorial testing is a "black-box test technique in which test cases are designed to exercise specific combinations of values of several parameters" (ISTQB). 

Test using random combination of parameter values

Random testing doesn't ensure we test combinations that matter.

...

Empyrical data

Several studies indicate that the vast majority of defects (67%-84%) related to input values are due to either to a problem in a parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).

Single-value faults are mostly probable to typical mistakes, such as the off-by-one bug (e.g., imagine using a loop and using the < operator instead of <=). The interaction of 2 parameters may be to bugs around implementing cascade conditional logic statements (e.g. using if or similar) involving those parameters/variables.

Bugs related to the interaction of more parameters decrease with the number of parameters; in other words, find these rare bugs will require much more tests to be performed, leading to more time/costs.

Pairwise and n-wise Testing

Given the empyrical data, adopting pairwise testing to test all the ombinations of pairs of parameters (sometimes also called as "all pairs testing") is a technique that is feasible.

Imagining the previous example, instead of having XXX test scenarios to perform, we would need just XX.

Sometimes, we may need to test more thoroughly some parameters, and for those we may choose to 3-way testing, for example, to ensure that we cover all the combinations of values of 3 relevant parameters.

Having a limited set of test generated, we can then execute them. However, usually algoritms generate these tests in a order, so that coverage is greater with the first tests and lesser with the last tests. This way, if we stop testing at a given moment, we can make sure that we track coverage and that we tested the most combinations possible.

In sum, there is a balance between the number of tests we execute and the coverage we will obtain.