Systems and features with variation

Systems Systems, in general, including software and hardware, usually have some input parameters that will affect the output(s) produced by the system. The number of parameters and the possible values, their ranges, vary and can be limited, huge but finite, or even infinite.

Taking a simple flight booking site as an example, we can easily have thousands of combinations for Flying From, Flying to, Class, (number of) Adults, and (number of) Children input parameters.

This potentially leads to a potential high - number of scenarios to be tested . For - for the previous example model above, and considering a model where parameters have just a with a limited number of possible values, that would still lead to 3*3*3*2*3=162 scenarios.

...

.

...

In general, we could think that if we aim to test systems like this in-depth, we would need to test the system them with all the possible combinations of values of these parameters.

But does it even make sense? Are any of those scenarios redundant, or in other words, is there any manageaable manageable subset of scenarios that can be tested that can still help us find bugs?

In this tutorial, we'll learn about the testing challenges of these systems and how to overcome them efficiently.

Initial testing options

Option 1: Test using

...

a “familiar” selection of values for the parameters

The first strategy that we may come up with would be adopting is data-driven testing. Data-driven testing It is a technique where a well-defined test script is executed multiple times, taking into account a "table" of parameters and corresponding values.

Usually, data-driven testing is used as a way to inject data to test automation scripts but . Still, it can also be used for to manually performing perform the same test multiple times against different data iterations.

The However, the exact combination of parameter values to be used is beyond of the scope of data-driven testing. However Usually, usually testers include parameter value combinations that represent examples coming as a direct consequence of acceptance criteria or , from well-known "happy paths," or from the production data.

Option 2: Test using a random combination of parameter values

Random testing is always an option, but it doesn't ensure we test combinations that matter unless we perform a very high number of tests, which would probabilistically include a certain % of combinations or even all of them if we spend an infinite time randomly testing.

Nobody wants to perform testing endlessly without any criteria. Random testing doesn't ensure we cover combinations that matter with a very manageable set of tests.

Option 3: Test every parameter/value combination

Testing every possible combination of parameters is only viable if we have very few parameters with very few possible values for each one of them.

In general, testing all combinations:

• takes considerable time
• is costly

...

• in terms of human resources or infrastructure

...

Test using random combination of parameter values

Random testing is always an option that comes but it doesn't ensure we test combinations that matter unless we perform a very high number of tests, which would probabilisticly include a certain % of combinations or even all of them if we spend an infinite time randomly testing.

Nobody wants to perform testing endlessly, without any sort of criteria. Random testing doesn't ensure we cover combinations that matter with a very limited set of tests.

Empyrical data

What empirical studies tell us about fault detection

Before we talk about the optimal testing option, let’s look at the underlying practical evidence.

Studies such as “Estimating t-Way Fault Profile Evolution During Testing” and “Practical Combinatorial Testing” (presented by the National Institute of Standards and Technology in 2017 and 2010 respectively) indicate Several studies indicate that the vast majority of defects (67%-84%93%) related to input values are due to either to a problem in a either one parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).

NIST research showed that most software bugs and failures are caused by one or two parameters, with progressively fewer by three or more, which means that combinatorial testing can provide more efficient fault detection than conventional methods. Multiple studies have shown fault detection equal to exhaustive testing with a 20X to 700X reduction in test set size.  New algorithms compressing combinations into a small number of tests have made this method practical for industrial use, providing better testing at lower cost. - NIST

The following chart (see reference ahead) represents the cumulative error detection rate for fault-triggering conditions. 

Image Added

The key insight underlying t-way combinatorial testing is that not every parameter contributes to every fault, and many faults are caused by interactions between a relatively small number of parameters - Combinatorial Software Testing, Rick Kuhn and Raghu Kacker, National Institute of Standards and Technology, Yu Lei, University of Texas at Arlington, Justin Hunter, Hexawise

Single-value faults are mostly probable most likely due to typical mistakes, such as the off-by-one bug (e.g., imagine using a loop and using the "<" operator instead of "<="). The interaction of 2 parameters may be due to bugs around implementing cascade conditional logic statements (e.g. using , using if or similar)  involving those parameters/ variables.

Bugs related to the interaction of more parameters decrease with the number of parameters; in other words, finding these rare bugs will require much more tests to be performed, leading to more time/costs.

...

However, those rare t-way interaction faults can also be critical and addressed by proper testing techniques.

Recommended Option: Combinatorial Testing considering 2-way (pairwise) and t-way

...

interaction of parameters

Given the empirical Given the empyrical data mentioned earlier, adopting pairwise algorithms to test all the combinations of pairs of parameters (sometimes also called as "all-pairs testing") is a technique that is not only feasible but also provides great results in terms of fault-detectioncombinatorial testing is an approach that provides great results in terms of fault detection/defect slippage prevention with a manageable test suite size.

It is a "black-box test technique in which test cases are designed to exercise specific combinations of values of several parameters" (ISTQB).

How do we come up with the exact combination of parameter values?

Instead of letting humans select them by hand, we rely on tools to perform them more efficiently. There are different algorithms for generating t-way interactions of parameters (e.g., pairs, triplets) - some may create more scenarios than others to achieve the same coverage (in terms of the interaction of parameters) and take more or less time. There are a couple of important algorithm features to consider, as seen ahead.

Reducing the number of test scenarios and the time to create them

The core function of combinatorial algorithms is identifying the smallest mathematically-possible set of scenarios to satisfy the t-way condition and doing that much faster than in the manual approach.

Imagining the previous example, instead of having 162 test scenarios to perform, we would need could have just 13 using the pairwise technique to generate them.

Sometimes, we may need to test more thoroughly some parameters, and for those we may opt for 3-way, for example, to ensure that we cover, e.g., all the combinations of 3 values of 3 relevant parameters. In this case we may have mixed-strength scenarios across all parameters, which would still require only 35 scenarios.

We may even have risk-based use cases where combinations of certain parameters may be are tested more thoroughly than others.

Having a limited set of test generated, we can then execute them. However, usually algoritms generate these tests in a order, so that Test scenarios are usually generated in a specific order so that the interaction coverage is greater with the first tests and lesser with the last testsones. This way, if we stop testing at a given moment, we can make sure ensure that we track coverage and that we tested the most combinations possible; some tools, such as Test Case Designer, allow us to track exactly the coverage of combinations achieved with a given number of executed tests.

In sum, there is a balance between the number of tests we execute and the coverage of interactions between variables (i.e., "t-way coverage") we will obtainvalidate.

Optimizing further the test scenarios to be performed

The first level of optimization is further reducing the number of generated test scenario. 

Combining business acumen with statistical techniques

Even if we use pairwise testing, or n-wise testing in general, to dramatically reduce the number of test scenarios, not all of these combinations may make sense for several reasons. The statistical side of the algorithm would not automatically account for the subject matter expertise.

For example, in our flight booking scenario the Departure and Destination parameters values need to be different in our flight booking scenario. Also, we may have some rules in place where , for example, the First class is not available unavailable to children.

These are restrictons that we can use in order to limit the generation of parameter combinations used by our test scenarioTherefore, the algorithm should support rule handling to limit specified interactions in the generated data set.

The second level of optimization is about including important scenarios first.

Not all combination Further, not all combinations of parameters may be equaly equally representative.

Sometimes there are parameter combinations we know that are highly important interactions as they represent highly frequently used happy paths , or whose business impact is high.or especially impactful previous defects.

The algorithm should allow users to intervene in the statistically-driven order and change the priority of certain scenariosWe can enforce these to appear in the generated scenarios and be the first ones.

Using pairwise and t-

...

way for scripted testing and exploratory testing

Whenever generating an optimized dataset (i.e., multiple "rows" of values for the parameters), this will be typically used to data-drive a scripted test case (e.g., a "manual" test composed of steps , or an automated test script).

In that case, testers would specify the steps to follow and include references to the parameters on of those steps. To perform testing, the test is iterated multiple times, as many as of the generated dataset rows (i.e., a combination of parameterparameters/values). In each iteration, the parameters are replaced by the corresponding values on the dataset row.

...

Generating these combinations is useful not only for this testing approach, though.

Pairwise and t-wise testing donway testing doesn't tell us how to actually perform testing; it just generates the combination of parameters. Therefore, we can also use this technique also if choose to we adopt a more exploratory testing approach, for example, for certain configurations of hardware/software.

Challenges

Pairwise, or t-wise way testing in general, even though useful, it's not is not a silver bullet.

Some challenges or limitations to be aware of include the following:

1. mindset shift: thinking about a system from the perspective of parameters, values, and their interactions is significantly different from traditional testing techniques. Therefore, combinatorial testing has a fairly steep learning curve, but the investment pays off in the medium term with improvements in both efficiency and quality.  
2. model scope: not all models are valuable, not all features have the same importance, and not all aspects of a feature have the same levels of risk. Even with the understanding of combinatorial methods, this testing approach requires significant collaboration between testers and business stakeholders to determine the right level of detail and priorities in each generated scenario (such collaboration should happen regardless, but its importance is increased in combinatorial and model-based testing);
3. test oracle: this this technique doesn't address finding the proper test oracle for the generated scenarios. How do we know the scenario is behaving as expected? How do we know that a given scenario has issues or not?modeling: depicting a "good" model requires the intervention of testers. Testers with the help of other team members are the ones able to figure out representative and important scenarios to model , their parameters, the values for those parameters, constraints, etc. 

Xray datasets and Xray Test Case Designer

Xray has built-in support for parameterized tests and datasets, supporting user-defined datasets and the automatic generation of combinations for the identified parameters.

Test Case Designer (TCD) provides a more comprehensive modeling tool , where it's possible to:

• define parameters and values,

...

• enforce system-under-test rules,
• generate optimized datasets

...

• using 2-way or

...

• t-way

...

• settings, up to a certain level, within minutes.

With Test Case Designer it's possible to , you can have a limited and manageabled manageable set of test scenarios to perform and make sure ensure that most combination combinations of parameters are met with the initial scenariosvalues are covered early on in the test suite, so that most risk is addressed upfront.

TCD doesn't replace Xray's built-in capabilities for parameterized tests and datasets; it's a more evolved approach. Both can be used in a given project.

...

...