Xray Cloud Documentation > Scenarios > XE_tier_feature.png

Introduction

The Xray Test Case Designer helps generate efficient Test scenarios by combining parameter values. Two key features in this process are Combination Strengths and Freeze Scenarios.

Combination Strengths control how many parameter values are tested together (e.g., 2-way, 3-way), allowing for optimized coverage based on your needs. Freeze Scenarios lets you lock specific scenarios, ensuring they remain as unchanged as possible even if parameters or settings are updated.

These features enhance Test coverage and preserve critical Test cases for consistent and reliable testing.

Combination Strengths

Combination Strengths refer to the depth or level of interaction between parameters that are considered when generating Test cases. It defines how many parameters are included together in the interaction coverage goal of the algorithm.

Use Case

In the next scenario, we focus on ordering a coffee, but there are many ways to order this drink. A barista, for instance, would need to be familiar with these combinations. Let’s look at what happens when the coverage strength is set to 1-way interactions (Figure 1 - 1).

Figure 1 - Dropdown

Only seven Tests are required to achieve our desired coverage.

This means that every Test condition was used at least once. This, however, does not guarantee that any parameter values were tested together. In other words, a parameter value merely appearing in this Test set is satisfactory as far as the Xray Test Case Designer is concerned. That’s not particularly thorough.

In this situation, the barista might not be prepared for any combination of particular requests when customers place their orders. Search for a combination of the values ‘Drink Type = Hot’ and ‘Drink Size = 20-ounce’ together in our set of seven scenarios. If this set of Tests were referenced solely, the barista would be wholly unprepared to make a hot 20-ounce coffee. Let’s make sure those two values are together.

When we create a set of 2-way interactions (Figure 2 - 1), 39 scenarios are required to achieve our coverage goal of testing every possible pair of Test conditions (Figure 2):

Figure 2 - Conditions

Although the quantity of Tests has increased, the quality and thoroughness of this set have drastically improved as well. By testing every possible pair of Test conditions, you have guaranteed that every value has been tested with all other parameters’ values.

A hot, 20-ounce drink is guaranteed to appear in your set of Tests. ‘Drink Type = Hot’ and ‘Drink Size = Short’ have been tested for. ‘Hot’ and any of the sizes listed have been tested for. Likewise, ‘Hot’ and any number of espresso shots have been tested. But this concept doesn’t just apply to ‘Hot’. It applies to ‘Frozen,’ ‘Decaf,’ ‘Grande,’ and any other parameter value in your set of inputs.

These examples go on, but there’s a catch: a hot, 20-ounce drink with espresso may not have been included. In 2-way interactions, only pairs have been tested. Enter the 3-way interactions (Figure 3 - 1).

221 scenarios achieve the goal of testing every possible “triplet” of Test conditions (Figure 3).

Figure 3 - Conditions

You’ve noticed the concerning increase in the number of Tests because of thoroughness. The more parameter values that you want to be tested together, the more tests there will be.

By increasing the coverage level to 3-way interactions (Figure 3 - 1), you have guaranteed that a hot, 20-ounce drink with any number of espresso shots has been included. The same goes for every other possible 3-way interaction in our set of Tests.

Let’s take this a step further; what if you absolutely needed a hot, 20-ounce drink with a specific number of espresso shots and decaf (or other caffeine type)? Although espresso shots with decaf in the same drink are unlikely in reality, people can have unusual tastes.

You may have also noticed the increased waiting time to generate more thorough scenarios. Not only is the number of Tests getting larger, but any margin of error in your system under Test decreases. This tradeoff between Test quantity and quality continues all the way up to 4, 5, and even 6-way interactions. This is because Tests will include every quadruplet, quintuplet, and sextuplet of test conditions, respectively.

Coverage Strengths

2-way (Pairwise)

Often a good default strength for many types of functional Tests.

3-way

Overall, could be considered in higher-risk / more “mission-critical” Testing. If you are executing Tests manually, rather than execute a complete set of 3-way scenarios, it is often preferable to execute a smaller set of “Mixed-Strength” Tests (with 3-way coverage focused on the most important interactions and 2-way coverage for the other parts).

4-way and Higher

Rarely used except in situations where:

All the Tests are automated (therefore cheap and quick to execute).
The potential risk for error could be extremely disastrous or expensive. For instance, safely sending people into space requires that no errors be made.

If you’re considering the pros and cons of, say, executing a smaller number of 2-way tests vs. a larger number of 3-way tests, it is extremely important for you to understand additional Test design strategies that you can use to decrease (or increase) the number of Tests Xray Test Case Designer generates.

To get a detailed explanation of the difference between less-thorough 2-way (AKA “pairwise” or “all pairs”) Tests and more-thorough n-way tests, you can hover over the i symbol (Figure 4 - 1), then click the "n-way interactions means..." (Figure 4 - 2). This will lead you to the screen depicted in Figure 5.

Figure 4 - Interactions

Figure 5 - Example

Figure 5 - Example Explanation

Freeze Scenarios

"Freeze Scenarios" refers to a feature that allows users to lock or "freeze" specific Test scenarios. Once frozen, these scenarios will not be affected by future changes in the Parameters, Constraints, or Test case generation settings. This is useful when certain Test scenarios are critical and must remain as unchanged as possible, even as other scenarios evolve or new ones are generated.

Use Case

Introduction

Here, you will learn how to ensure specific scenarios do not change as much even when you are updating your model.

What if we want to Test an airplane ticket reservation scenario, but need to edit it after completing an execution?

Let's assume this is the initial model state (Figures 6 and 7):

Figure 6 - Model

Figure 7 - Model

Operation

On the Scenarios screen, click the Freeze these Scenarios button (Figure 8 - 1).

Figure 8 - Freeze

You will see a warning message (Figure 9) reminding you that freezing your Test cases should only be done when there is a valid reason to do so.

To proceed, click the Freeze button (Figure 9 - 1).

Figure 9 - Freeze

Once you click Freeze (Figure 9 - 1), your existing Test cases will be “frozen” as Forced Interactions.

Here’s what that looks like now (Figure 10): all of our Tests have been “frozen” as the combinations of forced parameter values.

Figure 10 - Frozen

Let’s add another parameter: Amount of luggage (Figure 11 - 1). At this point, we may have already run some of these scenarios, so we don’t want to have to redo any of them, even if Xray Test Case Designer could reduce the total number of scenarios. By freezing the previous 11 Tests, Xray Test Case Designer will keep them as they are, and simply add a couple of scenarios to pair all of the new values – the numbers of luggage – with every other value.

Here are the new Tests (Figure 11). Note that the initial 11 scenarios are still present. We’ve simply added a few more.

Figure 11 - Tests

Here’s what the differences could have looked like if we had not frozen our initial 11 Tests.

Notice how Test Case 3 contained the following while frozen:

USA.
India.
Coach.
Selected.
1.
More than 1.
2 or more.

If we unfroze, Test Case 3 could contain the following:

USA.
India.
First.
Selected.
1.
0.
2 or more.

Therefore, we would have needed to rerun our initial scenarios. If we still want to unfreeze, we can do so by navigating to the Forced Interactions screen, and simply clicking the Unfreeze button in the top left.

“Optimized” compute option

Key points:

Selective Activation:
- The "optimized" compute option will not affect existing computations automatically.
- Users can select the "optimized" option from a dropdown menu on the Scenarios screen.
- To activate, users must not only select “optimized”, but also fill in a text field with the word “optimize” and confirm it. This intentional extra step ensures that the optimized compute is used sensibly.
Enhanced Computation Process:
- The "optimized" scenario option runs the computation on the same engine 50 times with a single click.
- It selects the outcome with the least number of tests, often resulting in slightly reduced scenario counts compared to the standard option.
- The non-deterministic variation is crucial for achieving this optimization.
Cache Management:
- Switching between standard and optimized options does reset the cache. For example, if you calculate using the standard option, switch to optimized, and then back to standard, a new data table will be generated, therefore, creating a new set of scenarios.
- Staying with the same compute option does not reset the cache. For instance, if you calculate using the optimized option, switch to another screen without making any cache-resetting edits, and then return to optimized, the same table will be retrieved.
Recommended Workflow:
- Utilize the standard scenario computation while the model is undergoing edits.
- Utilize the optimized scenario computation when you are ready to export the final results to include in test executions.

The examples of model actions that do break cache:

Adding/Removing parameters
Adding/Removing values
Adding/Removing/Editing constraints
Adding/Removing/Editing forced interactions
Copying the model/reverting to a different version of the model

The examples of model actions that do not break cache:

Editing value expansions
Renaming values
Navigating between tabs
Adding notes
Editing scripts

If you have questions or technical issues, please contact the Support team via the Customer Portal (Jira service management) or send us a message using the in-app chat.