Systems and features with some sort of variation

Systems in general, including software and hardware, usually have some input parameters that will affect the output(s) produced by the system. The number of parameters and the possible values, their ranges, vary and can be limited, huge but finite, or even infinite.
A system generates different results/outputs not only due to changes in the obvious input parameters but also due to using the system in different contexts (e.g., configurations, operating system, time zones, cloud provider). Those types of variation ideas should also be accounted for in your testing.
Taking a simple flight booking site as an example, we can easily have thousands of combinations for Flying From, Flying to, Class, (number of) Adults, (number of) Children input parameters.

...

Initial testing options

Option 1: Test using “familiar” selection of values for the parameters

The first strategy that we may come up with is data-driven testing. It is a technique where a well-defined test script is executed multiple times, taking into account a "table" of parameters and corresponding values.
Usually, data-driven testing is used as a way to inject data to test automation scripts but it can also be used to manually perform the same test multiple times against different data iterations.
However, the exact combination of parameter values to be used is beyond the scope of data-driven testing. Usually, testers include parameter value combinations that represent examples coming as a direct consequence of acceptance criteria, from well-known "happy paths", or from the production data.

...

Option 2: Test using random combination of parameter values

Random testing is always an option but it doesn't ensure we test combinations that matter unless we perform a very high number of tests, which would probabilistically include a certain % of combinations or even all of them if we spend an infinite time randomly testing.

Nobody wants to perform testing endlessly, without any sort of criteria. Random testing doesn't ensure we cover combinations that matter with a very manageable set of tests.

Option 3: Test every parameter/value combination

Testing every possible combination of parameters is only viable if we have very few parameters with very few possible values for each one of them.

...

Empirical data about fault detection

Studies, such as presented by NIST, PRACTICAL COMBINATORIAL TESTING, 2010,  indicate that the vast majority of defects (67%-93%) related to input values are due to either to a problem in a parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).

...

Bugs related to the interaction of more parameters decrease with the number of parameters; in other words, finding these rare bugs will require much more tests to be performed, leading to more time/costs. However, those rare t-way interaction faults can also be critical.

Combinatorial Testing considering 2-way (pairwise) and t-way interaction of parameters

Given the empirical data mentioned earlier, adopting combinatorial testing is an approach that provides great results in terms of fault detection/defect slippage prevention with manageable test suite size.

...

Instead of letting a human select them by hand, we rely on tools to perform it more efficiently. There are different algorithms for generating t-way interactions of parameters (e.g., pairs, triplets) - some may create more scenarios than others to achieve the same coverage (in terms of interaction of parameters), and take more or less time. There are a couple of important algorithm features to consider as seen ahead.

Reducing the number of test scenarios and the time to create them

The core function of combinatorial algorithms is identifying the smallest mathematically-possible set of scenarios to satisfy the t-way condition, and doing that much faster than in the manual approach.

...

In sum, there is a balance between the number of tests we execute and the coverage of interactions between variables (i.e., "t-way coverage") we validate.

Combining business acumen with statistical techniques

The first level of optimization is further reducing the number of generated test scenario. 

Even if we use pairwise testing, or n-wise testing in general, to dramatically reduce the number of test scenarios, not all of these combinations may make sense for several reasons. The statistical side of the algorithm would not automatically account for the subject matter expertise.

For example, in our flight booking scenario the Departure and Destination parameters values need to be different. Also, we may have some rules in place where , for example, the First class is not available to children.

These are restrictions that we can use in order to limit the generation of parameter combinations used by our test scenarioTherefore, the algorithm should support rule handling to limit specified interactions in the generated data set.

The second level of optimization is about including important scenarios first.

Not Further, not all combinations of parameters may be equally representative.

Sometimes there are parameter combinations we know that are interactions we consider highly important as they represent highly frequently used happy paths, or whose business impact is high.especially impactful previous defects.

The algorithm should allow users to intervene into the statistically-driven order and change the priority of certain scenariosWe can enforce these to appear in the generated scenarios and be the first ones.

Using pairwise and t-way for scripted testing and exploratory testing

Whenever generating an optimized dataset (i.e., multiple "rows" of values for the parameters) this will be typically used to data-drive a scripted test case (e.g., a "manual" test composed of steps, or an automated test script).

...

Pairwise and t-way testing don't tell us how to actually perform testing; it just generates the combination of parameters. Therefore, we can use this technique also if we choose to adopt a more exploratory testing approach, for example for certain configurations of hardware/software.

Challenges

Pairwise, or t-way testing in general, even though useful, is not a silver bullet.

Some challenges or limitations to be aware of, include:

1. mindset shift: thinking about a system from the perspective of parameters, values, and their interactions is significantly different from traditional testing techniques. Therefore, combinatorial testing has a fairly steep learning curve, but the investment pays off in the medium term with the improvements in both efficiency and quality.  
2. model scope: not all models are valuable, not all features have the same importance, not all aspects of a feature have the same levels of risk. Even with the understanding of combinatorial methods, this testing approach requires significant collaboration between testers and business stakeholders to determine the right level of detail and priorities in each generated scenario (such collaboration should happen regardless, but its importance is increased in combinatorial and model-based testing);
3. test oracle: this this technique doesn't address finding the proper test oracle for the generated scenarios. How do we know the scenario is behaving as expected? How do we know that a given scenario has issues or not?modeling: depicting a "good" model requires the intervention of testers. Testers with the help of other team members are the ones able to figure out representative and important scenarios to model , their parameters, the values for those parameters, constraints, etc. 

Xray datasets and Xray Test Case Designer

Xray has built-in support for parameterized tests and datasets, supporting user-defined datasets and automatic generation of combinations for the identified parameters.

Test Case Designer (TCD) provides a more comprehensive modeling tool, where it's possible to:

• define parameters and values,

...

• enforce system-under-test rules,
• generate optimized datasets

...

• using 2-way or t-way

...

• settings,up to a certain level, within minutes.

With Test Case Designer it's possible to you can have a limited and manageabled manageable set of test scenarios to perform and make sure that most combinations of parameters are met with the initial scenariosvalues are covered early on in the test suite, so that most risk is addressed upfront.

TCD doesn't replace Xray built-in capabilities for parameterized tests and datasets; it's a more evolved approach. Both can be used in a given project.

...

...