An explanation for how to best handle a common test design challenge related to dynamic cardinality.

The Problem

“Select All That Apply” scenarios occur frequently in web applications and are typically mishandled by test designers. Let’s look at the following example for some context:

...

Therefore, we need to account for pizzas that contain more than one topping in our tests. We’d also like to reduce the number of tests in our 2-way test set. How do we accomplish these goals? While there are multiple strategies you could employ to ensure that pizzas with multiple toppings to appear in your tests, let’s start with the most basic and widely-applicable solution.

First Recommended Strategy: Use Multiple Parameters (and Weight Them)

We want to start by including each of the available toppings as parameters. This will ensure that each and every topping can appear on a single pizza together. But here’s where it gets a bit tricky.

...

In a similar vein of thinking, you can adjust the weighting of each individual topping based on the frequency it will actually appear on a pizza. Obviously customers will choose Pepperoni as a topping more than they would order Hot Sauce, so it might makes sense to weight these toppings differently. For instance, you could apply a 2:1 ratio of “No” to “Yes” for Pepperoni, while increasing the ratio to 4:1 of “No” to “Yes” for Hot Sauce. This would result in the most repeatedly ordered toppings appearing more frequently in your tests without drastically increasing the total number of tests.

Second Strategy: Model a Maximum Number of Selections (Together with 1-Way/Mixed Strength Coverage)

If having an average of 6 “Select All That apply” items per test scenario seems excessive, this test strategy may be preferable.

...

One way to avoid this problem is to use mixed-strength test coverage, utilizing 1-way coverage for each of the pizza toppings while keeping the 2-way coverage for any additional parameters. Mixed-strength test models allow you to select different levels of coverage for different parameters. This allows you to control the balance between increased test thoroughness with the workload created by additional tests. If we simply adjust the coverage level of each of the 4 pizza toppings to 1-way coverage while keeping the rest of the parameters at 2-way coverage, we are left with only 13 scenarios.

Third Test Design Strategy: Model a Maximum Number of Selections (Together with the Value Expansion Feature)

As a way of keeping the number of tests you generate small, the Value Expansion feature often works very well with the “Modeling a Maximum Number of Selections” strategy. When using this strategy, Value Expansions are often a strong alternative to the strategy described immediately above of including the 1-way Mixed-Strength approach.

...

Please note that any time you use the Value Expansion feature, there will not be a guarantee that pairwise coverage will be achieved for all of the items in your Value Expansion lists.

Conclusion

The “Select All That Apply” pattern occurs frequently. Knowing how to weight various tradeoffs involved is important. Having a solid understanding of the pros and cons of different test design approaches that you can use in this situation will allow you to make healthy decisions. This article has referred to 6 strategies, and suggested that 3 are usually bad and 3 are usually worth considering.

...