Systems and features with some sort of variation

Systems in general, including software and hardware, usually have some input parameters that will affect the output(s) produced by the system. The number of parameters and the possible values, their ranges, vary and can be limited, huge but finite, or even infinite.

Taking a simple flight booking site as an example, we can easily have thousands of combinations for Flying From, Flying to, Class, (number of) Adults, (number of) Children input parameters.

This leads to a potential high-number of scenarios to be tested. For the previous example, and considering a model where parameters have just a limited number of possible values, that would still lead to 3*3*3*2*3=162 scenarios.

In general, we could think that if we aim to test systems like this in-depth, we would need to test the system with all the possible combinations of values of these parameters.

But does it even make sense? Are any of those scenarios redundant, or in other words, is there any manageaable subset of scenarios that can be tested that can still help us find bugs?

In this tutorial we'll learn about the testing challenges of these systems and how to overcome them efficiently.

Initial testing options

Test using some examples for the parameters

The first strategy that we may come with would be adopting data-driven testing.

Data-driven testing is a technique where a well-defined test script is executed multiple times, taking into account a "table" of parameters and corresponding values.

Usually, data-driven testing is used as a way to inject data to test automation scripts but it can also be used for manually performing the same test multiple times against different data.

The exact combination of parameter values to be used is beyond of the scope of data-driven testing. However, usually testers include parameter combinations that represent examples coming as a direct consequence of acceptance criteria or from well-known "happy paths".

Test every parameter/value combination

Testing every possible combination of parameters is only viable if we have very few parameters with very few possible values for each one of them.

In general, testing all combinations:

1. takes considerable time
2. is costly interms of human resources or infrastructure
3. may be innefficient (more info ahead)

Test using random combination of parameter values

Random testing is always an option that comes but it doesn't ensure we test combinations that matter unless we perform a very high number of tests, which would probabilisticly include a certain % of combinations or even all of them if we spend an infinite time randomly testing.

Nobody wants to perform testing endlessly, without any sort of criteria. Random testing doesn't ensure we cover combinations that matter with a very limited set of tests.

Empyrical data about fault detection

Studies, such as presented by NIST, PRACTICAL COMBINATORIAL TESTING, 2010,  indicate that the vast majority of defects (67%-93%) related to input values are due to either to a problem in a parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).

NIST research showed that most software bugs and failures are caused by one or two parameters, with progressively fewer by three or more, which means that combinatorial testing can provide more efficient fault detection than conventional methods. Multiple studies have shown fault detection equal to exhaustive testing with a 20X to 700X reduction in test set size.  New algorithms compressing combinations into a small number of tests have made this method practical for industrial use, providing better testing at lower cost. - NIST

The following chart (see reference ahead) represents the cumulative error detection rate for fault-triggering conditions. 

The key insight underlying t-way combinatorial testing is that not every parameter contributes to every fault and many faults are caused by interactions between a relatively small number of parameters - Combinatorial Software Testing, Rick Kuhn and Raghu Kacker, National Institute of Standards and Technology, Yu Lei, University of Texas at Arlington, Justin Hunter, Hexawise

Single-value faults are mostly probable to typical mistakes, such as the off-by-one bug (e.g., imagine using a loop and using the "<" operator instead of "<="). The interaction of 2 parameters may be to bugs around implementing cascade conditional logic statements (e.g. using if or similar) involving those parameters/variables.

Bugs related to the interaction of more parameters decrease with the number of parameters; in other words, finding these rare bugs will require much more tests to be performed, leading to more time/costs. However, those rare t-way interaction faults can also be critical.

Combinatorial Testing considering 2-way (pairwise) and t-way interaction of parameters

Combinatorial testing is at first sight a technique that seems adequate precisely for this purpose, as the name shows.

How do we come with the exact combination of parameter values?

We could generate all possible combinations as mentioned earlier. However, and given the empyrical data mentioned earlier, adopting 2-way (pairwise) algorithms to test all the combinations of pairs of parameters (sometimes also called as "all-pairs testing") is a technique that provides great results in terms of fault-detection with a limited set of tests.

Reducing the number of test scenarios

Imagining the previous example, instead of having 162 test scenarios to perform, we could have just 13 using pairwise to generate them.

Sometimes, we may need to test more thoroughly some parameters together, and for those we may choose 3-way interactions, for example, to ensure that we cover all the combinations of values of 3 relevant parameters. We may even have mixed-strength scenarios where combinations of certain parameters are tested more thoroughly than others.

Test scenarios are usually generated in a order, so that coverage is greater with the first tests and lesser with the last ones. This way, if we stop testing at a given moment, we can make sure that we tested the most combinations possible; we can track exactly the coverage of combinations with the number of executed tests.

In sum, there is a balance between the number of tests we execute and the coverage of interactions between variables (i.e., "t-way coverage") we validate.

Optimizing further the test scenarios to be performed

The first level of optimization is further reducing the number of generated test scenario. 

Even if we use pairwise testing, or n-wise testing in general, to dramatically reduce the number of test scenarios, not all of these combinations may make sense for several reasons.

For example, in our flight booking scenario the Departure and Destination parameters need to be different. Also, we may have some rules in place where, for example, the First class is not available to children.

These are restrictons that we can use in order to limit the generation of parameter combinations used by our test scenario.

The second level of optimization is about including important scenarios first.

Not all combination of parameters may be equaly representative. Sometimes there are parameter combinations we know that are highly important as they represent highly used happy paths, or whose business impact is high.

We can enforce these to appear in the generated scenarios and be the first ones.

Using pairwise and t-way for scripted testing and exploratory testing

Whenever generating an optimized dataset (i.e., multiple "rows" of values for the parameters) this will be typically used to data-drive a scripted test case (e.g., a "manual" test composed of steps, or an automated test script).

In that case, testers would specify the steps to follow and include references to the parameters on those steps. To perform testing, the test is iterated multiple times, as many as of the generated dataset rows (i.e. combination of parameter/values). In each iteration the parameters are replaced by the corresponding values on the dataset row.

Generating these combinations is useful not only for this testing approach though.

Pairwise and t-way testing don't tell us how to actually perform testing; it just generates the combination of parameters. Therefore, we can use this technique also if choose to adopt a more exploratory testing approach, for example for certain configurations of hardware/software.

Challenges

Pairwise, or t-way testing in general, even though useful, is not a silver bullet.

Some challenges or limitations to be aware of, include:

1. test oracle: this technique doesn't address finding the proper test oracle for the generated scenarios. How do we know the scenario is behaving as expected? How do we know that a given scenario has issues or not?
2. modeling: depicting a "good" model requires the intervention of testers. Testers with the help of other team members are the ones able to figure out representative and important scenarios to model , their parameters, the values for those parameters, constraints, etc. 

Xray datasets and Xray Test Case Designer

Xray has built-in support for parameterized tests and datasets, supporting user-defined datasets and automatic generation of combinations for the identified parameters.

Test Case Designer (TCD) provides a more comprehensive modeling tool, where it's possible to define parameters, apply constraints, enforce specific scenarios, generate optimized datasets/scenarios using 2-way or t-way to ensure that certain parameters are tested together, up to a certain level. With Test Case Designer it's possible to have a limited and manageabled set of test scenarios to perform and make sure that most combination of parameters are met with the initial scenarios, so that most risk is addressed upfront.

TCD doesn't replace Xray built-in capabilities for parameterized tests and datasets; it's a more evolved approach. Both can be used in a given project.