Overview

Systems in general, including software and hardware, usually have some input parameters that will affect the output produced by the system. The number of parameters and the possible values, their ranges, vary and can be limited, huge but finite, or even infinite.

Taking a simple flight booking site as an example, we can easily have thousands of combinations for Flying From, Flying to, Class, (number of) Adults, (number of) Children input parameters.

It's possible to remove some values of the combinations to be generated. For example, we can exclude the "First" Class. That would lead to less scenarios to test (e.g., 162 => 108).

This leads to a potential high-number of scenarios to be tested. For the previous example, and considering a model where parameters have just a limited number of possible values, that would still lead to 3*3*3*2*3=162 scenarios.

IIs it feasible? Does it even make sense? Are any of those scenarios redundant, or in other words, is there any manageaable subset of scenarios that can be tested that can still help us find bugs?

In this tutorial we'll learn about the testing challenges of these systems and how to overcome them efficiently.

Initial testing options

Test some well-known values for parameters

The first strategy would be on adopting data-driven testing.

Data-driven testing is a technique where a well-defined test script is executed multiple times, taking into account a "table" of parameters and corresponding values.

Usually, data-driven testing is used as a way to inject data to test automation scripts but it can also be used for manually performing the same test multiple times against different data.

The exact combination of parameter values to be used is beyond of the scope of data-driven testing.

Test every parameter combination

At first sight, if we aim to test this system in-depth, we would need to test the system with all the possible combinations of values of these parameters.

This would:

1. take long time
2. be costly
3. be eventually innefficient (more info ahead)

Combinatorial testing is a "black-box test technique in which test cases are designed to exercise specific combinations of values of several parameters" (ISTQB). 

Test using random combination of parameter values

Random testing doesn't ensure we test combinations that matter.

...

Empyrical data

Several studies indicate that the vast majority of defects (67%-84%) related to input values are due to either to a problem in a parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).

Single-value faults are mostly probable to typical mistakes, such as the off-by-one bug (e.g., imagine using a loop and using the < operator instead of <=). The interaction of 2 parameters may be to bugs around implementing cascade conditional logic statements (e.g. using if or similar) involving those parameters/variables.

Bugs related to the interaction of more parameters decrease with the number of parameters; in other words, find these rare bugs will require much more tests to be performed, leading to more time/costs.

Pairwise and n-wise Testing

Given the empyrical data, adopting pairwise testing to test all the ombinations of pairs of parameters (sometimes also called as "all pairs testing") is a technique that is not only feasible but also provides great results in terms of fault-detection.

Imagining the previous example, instead of having XXX test scenarios to perform, we would need just XX.

Sometimes, we may need to test more thoroughly some parameters, and for those we may choose to 3-way testing, for example, to ensure that we cover all the combinations of values of 3 relevant parameters. In this case we may have mixed-strength scenarios where combinations of certain parameters may be tested more thoroughly than others.

Having a limited set of test generated, we can then execute them. However, usually algoritms generate these tests in a order, so that coverage is greater with the first tests and lesser with the last tests. This way, if we stop testing at a given moment, we can make sure that we track coverage and that we tested the most combinations possible.

In sum, there is a balance between the number of tests we execute and the coverage we will obtain.

Not all combinations make sense for several reasons.

For example, in our flight booking scenario the Departure and Destination parameters need to be different. Also, we may have some rules in place where, for example, the First class is not available to children.

These are restrictons that we can use in order to limit the generation of parameter combinations used by our test scenario.

But not all combination of parameters may be equaly representative. Sometimes there are parameter combinations we know that are highly important as they represent highly used happy paths, or whose business impact is high.

We can enforce these to appear in the generated scenarios and be the first ones.