Versions Compared

Old Version 1

changes.mady.by.user Helder Ferreira

Saved on

compared with

New Version 2

changes.mady.by.user Ana Duarte

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This section covers the basics of understanding the Xray Test Case Designer intelligent augmentation algorithm and settings.


UI Expand
titleTable of Contents

Table of Contents

Introduction

The Xray Test Case Designer helps generate efficient Test scenarios by combining parameter values. Two key features in this process are Combination Strengths and Freeze Scenarios.

Combination Strengths control how many parameter values are tested together (e.g., 2-way, 3-way), allowing for optimized coverage based on your needs. Freeze Scenarios lets you lock specific scenarios, ensuring they remain as unchanged as possible even if parameters or settings are updated.

These features enhance Test coverage and preserve critical Test cases for consistent and reliable testing.

Combination Strengths

Combination Strengths refer to the depth or level of interaction between parameters that are considered when generating Test cases. It defines how many parameters are included together in the interaction coverage goal of the algorithm.

Use Case

In the next scenario, we focus on ordering a coffee, but there are many ways to order this drink. A barista, for instance, would need to be familiar with these combinations. Let’s look at what happens when the coverage strength is set to 1-way interactions (Figure 1 - 1).

Figure 1 - DropdownImage Added

Figure 1 - Dropdown


Info

Only seven Tests are required to achieve our desired coverage.


This means that every Test condition was used at least once. This, however, does not guarantee that any parameter values were tested together. In other words, a parameter value merely appearing in this Test set is satisfactory as far as the Xray Test Case Designer is concerned. That’s not particularly thorough.

In this situation, the barista might not be prepared for any combination of particular requests when customers place their orders. Search for a combination of the values ‘Drink Type = Hot’ and ‘Drink Size = 20-ounce’ together in our set of seven scenarios. If this set of Tests were referenced solely, the barista would be wholly unprepared to make a hot 20-ounce coffee. Let’s make sure those two values are together.

When we create a set of 2-way interactions (Figure 2 - 1), 39 scenarios are required to achieve our coverage goal of testing every possible pair of Test conditions (Figure 2):

Figure 2 - ConditionsImage Added

Figure 2 - Conditions


Although the quantity of Tests has increased, the quality and thoroughness of this set have drastically improved as well. By testing every possible pair of Test conditions, you have guaranteed that every value has been tested with all other parameters’ values.

A hot, 20-ounce drink is guaranteed to appear in your set of Tests. ‘Drink Type = Hot’ and ‘Drink Size = Short’ have been tested for. ‘Hot’ and any of the sizes listed have been tested for. Likewise, ‘Hot’ and any number of espresso shots have been tested. But this concept doesn’t just apply to ‘Hot’. It applies to ‘Frozen,’ ‘Decaf,’ ‘Grande,’ and any other parameter value in your set of inputs.

These examples go on, but there’s a catch: a hot, 20-ounce drink with espresso may not have been included. In 2-way interactions, only pairs have been tested. Enter the 3-way interactions (Figure 3 - 1).

221 scenarios achieve the goal of testing every possible “triplet” of Test conditions (Figure 3).

Figure 3 - ConditionsImage Added

Figure 3 - Conditions


You’ve noticed the concerning increase in the number of Tests because of thoroughness. The more parameter values that you want to be tested together, the more tests there will be.

By increasing the coverage level to 3-way interactions (Figure 3 - 1), you have guaranteed that a hot, 20-ounce drink with any number of espresso shots has been included. The same goes for every other possible 3-way interaction in our set of Tests.

Let’s take this a step further; what if you absolutely needed a hot, 20-ounce drink with a specific number of espresso shots and decaf (or other caffeine type)? Although espresso shots with decaf in the same drink are unlikely in reality, people can have unusual tastes.

You may have also noticed the increased waiting time to generate more thorough scenarios. Not only is the number of Tests getting larger, but any margin of error in your system under Test decreases. This tradeoff between Test quantity and quality continues all the way up to 4, 5, and even 6-way interactions. This is because Tests will include every quadruplet, quintuplet, and sextuplet of test conditions, respectively.

Coverage Strengths

2-way (Pairwise)

Often a good default strength for many types of functional Tests.

3-way

Overall, could be considered in higher-risk / more “mission-critical” Testing. If you are executing Tests manually, rather than execute a complete set of 3-way scenarios, it is often preferable to execute a smaller set of “Mixed-Strength” Tests (with 3-way coverage focused on the most important interactions and 2-way coverage for the other parts).

4-way and Higher

 Rarely used except in situations where:

  •  All the Tests are automated (therefore cheap and quick to execute).
  •  The potential risk for error could be extremely disastrous or expensive. For instance, safely sending people into space requires that no errors be made.

If you’re considering the pros and cons of, say, executing a smaller number of 2-way tests vs. a larger number of 3-way tests, it is extremely important for you to understand additional Test design strategies that you can use to decrease (or increase) the number of Tests Xray Test Case Designer generates.

Info

To get a detailed explanation of the difference between less-thorough 2-way (AKA “pairwise” or “all pairs”) Tests and more-thorough n-way tests, you can hover over the i symbol (Figure 4 - 1), then click the "n-way interactions means..." (Figure 4 - 2). This will lead you to the screen depicted in Figure 5.

Figure 4 - InteractionsImage Added

Figure 4 - Interactions


Figure 5 - ExampleImage Added

Figure 5 - Example Explanation

Freeze Scenarios

"Freeze Scenarios" refers to a feature that allows users to lock or "freeze" specific Test scenarios. Once frozen, these scenarios will not be affected by future changes in the Parameters, Constraints, or Test case generation settings. This is useful when certain Test scenarios are critical and must remain as unchanged as possible, even as other scenarios evolve or new ones are generated.

Use Case

Introduction

Here, you will learn how to ensure specific scenarios do not change as much even when you are updating your model.

What if we want to Test an airplane ticket reservation scenario, but need to edit it after completing an execution?

Let's assume this is the initial model state (Figures 6 and 7):

Figure 6 - ModelImage Added

Figure 6 - Model


Figure 7 - ModelImage Added

Figure 7 - Model

Operation

UI Steps
UI Step

On the Scenarios screen, click the Freeze these Scenarios button (Figure 8 - 1).

Figure 8 - FreezeImage Added

Figure 8 - Freeze

UI Step

You will see a warning message (Figure 9) reminding you that freezing your Test cases should only be done when there is a valid reason to do so.

To proceed, click the Freeze button (Figure 9 - 1).

Figure 9 - FreezeImage Added

Figure 9 - Freeze

UI Step

Once you click Freeze (Figure 9 - 1), your existing Test cases will be “frozen” as Forced Interactions.

Here’s what that looks like now (Figure 10): all of our Tests have been “frozen” as the combinations of forced parameter values.

Figure 10 - FrozenImage Added

Figure 10 - Frozen

UI Step

Let’s add another parameter: Amount of luggage (Figure 11 - 1). At this point, we may have already run some of these scenarios, so we don’t want to have to redo any of them, even if Xray Test Case Designer could reduce the total number of scenarios. By freezing the previous 11 Tests, Xray Test Case Designer will keep them as they are, and simply add a couple of scenarios to pair all of the new values – the numbers of luggage – with every other value.

UI Step

Here are the new Tests (Figure 11). Note that the initial 11 scenarios are still present. We’ve simply added a few more.

Figure 11 - TestsImage Added

Figure 11 - Tests


Here’s what the differences could have looked like if we had not frozen our initial 11 Tests. 

Notice how Test Case 3 contained the following while frozen:

  • USA.
  • India.
  • Coach.
  • Selected.
  • 1.
  • More than 1.
  • 2 or more.

If we unfroze, Test Case 3 could contain the following:

  • USA.
  • India.
  • First.
  • Selected.
  • 1.
  • 0.
  • 2 or more.

Therefore, we would have needed to rerun our initial scenarios. If we still want to unfreeze, we can do so by navigating to the Forced Interactions screen, and simply clicking the Unfreeze button in the top left.


Expand
titleSupport/Troubleshooting

If you have questions or technical issues, please contact the Support team via the Customer Portal (Jira service management) or send us a message using the in-app chat.

...