Systems and features with some sort of variation

Systems Systems, in general, including software and hardware, usually have some input parameters that will affect the output(s) produced by the system. The number of parameters and the possible values, their ranges, vary and can be limited, huge but finite, or even infinite.A system generates different results/outputs not only due to changes in the obvious input parameters but also due to using the system in different contexts (e.g., configurations, operating system, time zones, cloud provider). Those types of variation ideas should also be accounted for in your testing.

Taking a simple flight booking site as an example, we can easily have thousands of combinations for Flying From, Flying to, Class, (number of) Adults, and (number of) Children input parameters.

...

This potentially leads to a high number of scenarios to be tested - for the example model above, with  with a limited number of possible values, that would still lead to 3*3*3*2*3=162 scenarios.

...

But does it even make sense? Are any of those scenarios redundant, or in other words, is there any manageable subset of scenarios that can be tested that can still help us find bugs?

In this tutorial, we'll learn about the testing challenges of these systems and how to overcome them efficiently.

...

Initial testing options

Option 1: Test using a “familiar” selection of values for the parameters

The first strategy that we may come up with is data-driven testing. It is a technique where a well-defined test script is executed multiple times, taking into account a "table" of parameters and corresponding values.

Usually, data-driven testing is used as a way to inject data to test automation scripts but . Still, it can also be used to manually perform the same test multiple times against different data iterations.

However, the exact combination of parameter values to be used is beyond the scope of data-driven testing. Usually, testers include parameter value combinations that represent examples coming as a direct consequence of acceptance criteria, from well-known "happy paths," , or from the production data.

...

Option 2: Test using a random combination of parameter values

Random testing is always an option, but it doesn't ensure we test combinations that matter unless we perform a very high number of tests, which would probabilistically include a certain % of combinations or even all of them if we spend an infinite time randomly testing.

Nobody wants to perform testing endlessly , without any sort of criteria. Random testing doesn't ensure we cover combinations that matter with a very manageable set of tests.

...

...

What empirical studies tell us about fault detection

Before we talk about the optimal testing option, let’s look at the underlying practical evidence.

Studies , such as presented by NIST, PRACTICAL COMBINATORIAL TESTING, 2010,  indicate “Estimating t-Way Fault Profile Evolution During Testing” and “Practical Combinatorial Testing” (presented by the National Institute of Standards and Technology in 2017 and 2010 respectively) indicate that the vast majority of defects (67%-93%) related to input values are due to either to a problem in a either one parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).

...

The key insight underlying t-way combinatorial testing is that not every parameter contributes to every fault, and many faults are caused by interactions between a relatively small number of parameters - Combinatorial Software Testing, Rick Kuhn and Raghu Kacker, National Institute of Standards and Technology, Yu Lei, University of Texas at Arlington, Justin Hunter, Hexawise

Single-value faults are mostly probable most likely due to typical mistakes, such as the off-by-one bug (e.g., imagine using a loop and using the "<" operator instead of "<="). The interaction of 2 parameters may be due be due to bugs around implementing cascade conditional logic statements (e.g. using , using if or similar)  involving those parameters/ variables.

Bugs related to the interaction of more parameters decrease with the number of parameters; in other words, finding these rare bugs will require much more tests to be performed, leading to more time/costs. However, those rare t-way interaction faults can also be critical and addressed by proper testing techniques.

Recommended Option: Combinatorial Testing considering 2-way (pairwise) and t-way interaction of parameters

Given the empirical data mentioned earlier, adopting combinatorial testing is an approach that provides great results in terms of fault detection/defect slippage prevention with a manageable test suite size.

It is a "black-box test technique in which test cases are designed to exercise specific combinations of values of several parameters" (ISTQB).

How do we come up with the exact combination of parameter values?

Instead of letting a human humans select them by hand, we rely on tools to perform it them more efficiently. There are different algorithms for generating t-way interactions of parameters (e.g., pairs, triplets) - some may create more scenarios than others to achieve the same coverage (in terms of the interaction of parameters) , and take more or less time. There are a couple of important algorithm features to consider, as seen ahead.

Reducing the number of test scenarios and the time to create them

The core function of combinatorial algorithms is identifying the smallest mathematically-possible set of scenarios to satisfy the t-way condition , and doing that much faster than in the manual approach.

...

Test scenarios are usually generated in a specific order , so that the interaction coverage is greater with the first tests and lesser with the last ones. This way, if we stop testing at a given moment, we can make sure ensure that we tested the most combinations possible; some tools, such as Test Case Designer, allow us to track exactly the coverage of combinations achieved with a given number of executed tests.

...

Even if we use pairwise testing, or n-wise testing in general, to dramatically reduce the number of test scenarios, not all of these combinations may make sense for several reasons. The statistical side of the algorithm would not automatically account for the subject matter expertise.

For example, in our flight booking scenario the Departure and Destination values need to be different in our flight booking scenario. Also, we may have some rules in place where the First class is not available unavailable to children.

Therefore, the algorithm should support rule handling to limit specified interactions in the generated data set.

...

...

Further, not all combinations of parameters may be equally representative.

Sometimes there are interactions we consider highly important interactions as they represent frequently used happy paths , or especially impactful previous defects.

The algorithm should allow users to intervene into in the statistically-driven order and change the priority of certain scenarios.

...

...

Whenever generating an optimized dataset (i.e., multiple "rows" of values for the parameters), this will be typically used to data-drive a scripted test case (e.g., a "manual" test composed of steps , or an automated test script).

In that case, testers would specify the steps to follow and include references to the parameters on of those steps. To perform testing, the test is iterated multiple times, as many as of the generated dataset rows (i.e., a combination of parameterparameters/values). In each iteration, the parameters are replaced by the corresponding values on the dataset row.

...

Generating these combinations is useful not only for this testing approach, though.

Pairwise and t-way testing dondoesn't tell us how to actually perform testing; it just generates the combination of parameters. Therefore, we can also use this technique also if we choose to adopt a more exploratory testing approach, for example, for certain configurations of hardware/software.

...

Some challenges or limitations to be aware of , include the following:

1. mindset shift: thinking about a system from the perspective of parameters, values, and their interactions is significantly different from traditional testing techniques. Therefore, combinatorial testing has a fairly steep learning curve, but the investment pays off in the medium term with the improvements in both efficiency and quality.  
2. model scope: not all models are valuable, not all features have the same importance, and not all aspects of a feature have the same levels of risk. Even with the understanding of combinatorial methods, this testing approach requires significant collaboration between testers and business stakeholders to determine the right level of detail and priorities in each generated scenario (such collaboration should happen regardless, but its importance is increased in combinatorial and model-based testing);
3. test oracle: this technique doesn't address finding the proper test oracle for the generated scenarios. How do we know the scenario is behaving as expected? How do we know that a given scenario has issues or not?

...

Xray has built-in support for parameterized tests and datasets, supporting user-defined datasets and the automatic generation of combinations for the identified parameters.

Test Case Designer (TCD) provides a more comprehensive modeling tool , where it's possible to:

• define parameters and values,
• enforce system-under-test rules,
• generate optimized datasets using 2-way or t-way settings, up to a certain level, within minutes.

With Test Case Designer, you can have a manageable set of test scenarios to perform and make sure ensure that most combinations of values are covered early on in the test suite, so that most risk is addressed upfront.

TCD doesn't replace Xray's built-in capabilities for parameterized tests and datasets; it's a more evolved approach. Both can be used in a given project.

...

...