Overview

Xray Test Case Designer allows to use different coverage settings, learn the strategies behind using it.

In the next scenario we focus on ordering a coffee, but there are many ways to order this beloved drink.
A barista, for instance, would need to be familiar with these combinations. Let’s look at what happens when the coverage strength is set to “1-way interactions" (using the dropdown at the top of the Scenarios screen).

Only seven tests are required to achieve our desired coverage

This means that every test condition was used at least once. This, however, does not guarantee that any parameter values were tested together. In other words, a parameter value merely appearing in this test set is satisfactory as far as Xray Test Case Designer is concerned. That’s not particularly thorough.

In this situation, our barista friend might not be prepared for any combination of particular requests when customers place their orders. Search for a combination of the values ‘Drink Type = Hot’ and ‘Drink Size = 20-ounce’ together in our set of 7 scenarios. If solely this set of tests were referenced, the barista would be wholly unprepared to make a hot 20-ounce coffee. Let’s make sure those two values are together!

When we create a set of 2-way interactions, 39 scenarios are required to achieve our coverage goal of testing every possible PAIR of test conditions:

Although the quantity of tests has increased, the quality and thoroughness of this set has drastically improved as well. By testing every possible pair of test conditions, you have guaranteed that every value has been tested with all other parameters’ values.

A hot, 20-ounce drink is guaranteed to appear in your set of tests. ‘Drink Type = Hot’ and ‘Drink Size = Short’ has been tested for. ‘Hot’ and any of the sizes listed have been tested for. Likewise, ‘Hot’ and any number of espresso shots has been tested. But this concept doesn’t just apply to ‘Hot’. It applies to ‘Frozen,’ ‘Decaf,’ ‘Grande,’ and any other parameter value in your set of inputs.

These examples go on, but there’s a catch: a hot, 20-ounce drink with espresso may not have been included. In 2-way interactions, only pairs have been tested. Enter the 3-way interactions.

221 scenarios achieve the goal of testing every possible “triplet” of test conditions

We’re sure you’ve noticed the concerning increase in the number of tests. Thoroughness comes with a price. The more parameter values that you want to be tested together, the more tests there will be.

By increasing the coverage level to “3-way interactions,”, you have guaranteed that a hot, 20-ounce drink with any number of espresso shots has been included. The same goes for every other possible 3-way interaction in our set of tests.

Let’s take this a step further; what if you absolutely needed a hot, 20-ounce drink with a specific number of espresso shots and decaf (or other caffeine type)? Although espresso shots with decaf in the same drink is unlikely in reality, people can have unusual tastes.

You may have also noticed the increased wait time to generate more-thorough scenarios. Not only is the number of tests getting larger, but any margin of error in your System under Test decreases. As you may have guessed, this tradeoff between test quantity and quality continues all the way up to 4-, 5-, and even 6-way interactions. This is because tests will include every quadruplet, quintuplet, and sextuplet of test conditions, respectively.

• 2-way (pairwise) coverage strength

  is often a good default strength for many types of functional tests
  
  

• 3-way coverage strength

   overall, could be considered in higher-risk / more “mission-critical” testing. If you are executing tests by hand, rather than execute a complete set of 3-way scenarios, it is often preferable to execute a smaller set of “Mixed-Strength” tests (with 3-way coverage focused on the most important interactions and 2-way coverage for the other parts).
  
  

• 4-way and higher coverage strengths

   are rarely used except in situations where (a) all the tests are automated (therefore cheap and quick to execute) and/or (b) the potential risk for error could be extremely disastrous or expensive. For instance, safely sending people into space requires that no errors be made.

If you’re considering the pros and cons of, say, executing a smaller number of 2-way tests vs. a larger number of 3-way tests, it is extremely important for you to understand additional test design strategies that you can use to decrease (or increase) the number of tests Xray Test Case Designer generates.