Introduction
The Analysis feature helps you to understand specific interaction coverage after every Scenario. It ensures proper Test coverage, identifies less impactful Test cases, and optimizes Test selection for efficiency.
By using Analysis, you can make data-driven decisions to improve quality while minimizing risk and resource use.
Usage
Coverage Matrix
Coverage Matrix refers to a method of evaluating the completeness of your Test Scenarios by analyzing the combinations of Parameter values. It ensures that each pair of Parameters is adequately tested according to the specified combination strength (i.e., 2-way).
In the Xray Test Case Designer, click My Test Models (Figure 1 - 1) and then click a Test model (Figure 1 - 2).
Figure 1 - Test model
A menu will open on the left. There, click Analysis (Figure 2 - 1) and select Coverage Matrix (Figure 2 - 2).
Figure 2 - Analysis
Understanding the Matrix (Figure 3)
Each square in the matrix chart represents a value pair. You can hover over the elements of the matrix to see more details (such as parameter names, status, etc.). Here's how to read the chart:
- Read the values listed across the top.
- Read the values listed on the left side.
- Find the intersection of two values (one from the top and one from the side):
- Green: the pair is already covered.
- Red: the pair is not yet covered, based on the scenario selected via the slider.
- Black: the pair is marked as invalid.
Figure 3 - Matrix
Sliding Your Way to Precise Coverage
The slider (Figure 2 - 4) allows you to visualize which pairs are covered in each Test.
When you arrive at the Matrix chart screen, the chart will automatically cycle through Test Case 0 to Test Case N (the final test in your set). The slider works as follows: less coverage to the left and more coverage to the right.
Communicating your Testing Coverage Precisely
The Matrix chart allows you to communicate your interaction coverage more easily and precisely.
Imagine a situation where you've created some Tests, and your manager says, “Development is running behind, so we’ll only have time to execute two-thirds of your Tests before going live. What will be missed?” You can easily respond with a slide like the one in Figure 4.
Figure 4 - Coverage scenarios
With this information, you can weigh your testing options. Do you need to test for a 2-way interaction that remains untested by Test 10? Is it worth generating six more Test cases? Or would it be better to add a forced interaction to prioritize that pair? The coverage matrix can be invaluable in the decision-making process.
- Can I See 3-way and Higher Interactions?
Currently, the Matrix chart only supports showing the pairs covered in each test.
- Note for Color-blind Users
Pressing “C” on your keyboard changes the color scheme of the Matrix chart. The following three options are available: Red/Green/Black, Orange/Blue/Black, Purple/Orange/Black. - Dealing with a Large Number of Values
Having 50+ total values can appear to be overwhelming in the Matrix Chart. Xray Test Case Designer can help make such use cases more manageable with Zoom and click-drag features:- Each matrix cell has a pop-up window with the details in larger font (compared to value names).
- Scrolling the mouse wheel while your cursor hovers over the matrix will increase or decrease the zoom of the section in the view.
- In the zoomed-in state, you can left-click the matrix, hold & drag to bring a different segment into visibility.
You can export the coverage data into a JSON or XML file by selecting the respective option from the dropdown menu (Figure 2 - 3). The file will be immediately saved to your machine, and you can easily share it with others.
Coverage Graph
Graph Coverage assesses the flow and transitions between different Test scenarios or states. It ensures that all possible paths or sequences through a process are tested, particularly focusing on state-based or workflow-driven Tests. This type of coverage is useful for identifying gaps in the logic or flow of complex systems and ensuring that all transitions are properly validated.
In the Xray Test Case Designer, click My Test Models (Figure 6 - 1) and then click a Test model (Figure 6 - 2).
A menu will open on the left. There, click Analysis (Figure 7 - 4) and select Coverage Graph (Figure 7 - 5).
Figure 7 - Analysis
The Coverage Graph screen will open (Figure 7).
Here, you can apply the Graph to any generated strength (Figure 7 - 1), but it will only display the cumulative coverage percentage, not the specific interactions.
The shape of the curve (Figure 7 - 3) is important to note: the Xray Test Case Designer algorithm frontloads as much variation as possible. Therefore, diminishing returns on incremental coverage improvement are reached fairly quickly, and the scenarios in the later part of the test suite contribute significantly less compared to the first few. The order is based on the Index column from the Scenarios screen. Sorting the data table differently does not affect the Graph or Matrix, as they always display the recommended execution priority. If you want to analyze the modified Scenarios table, you must either freeze it or manually re-import it as Forced Interactions.
You can hover over any point on the curve to see the corresponding percentage of interactions tested (Figure 7 - 3).
Example
The Analysis coverage charts can be extremely useful in answering questions like: "How much coverage is each of my Tests adding?" and "How much testing is enough?"
It may take a few minutes to understand the valuable information presented in the charts. The number of Parameters and Values entered in the Parameters screen determines how many total possible pairs of values exist in your Test model.
See the example with eight values (Figure 7):
Figure 7 - Values
Given these inputs, your model will have exactly 24 possible combinations of pairs of values, as shown below:
| Total pairs to be tested (24) | |||||
|---|---|---|---|---|---|
| Large & Heavy | Small & Heavy | ||||
| Large & Light | Small & Light | ||||
| Large & Purple | Small & Purple | Heavy & Purple | Light & Purple | ||
| Large & Green | Small & Green | Heavy & Green | Light & Green | ||
| Large & Hexagon | Small & Hexagon | Heavy & Hexagon | Light & Hexagon | Purple & Hexagon | Green & Hexagon |
| Large & Circle | Small & Circle | Heavy & Circle | Light & Circle | Purple & Circle | Green & Circle |
The first Test case (Large / Heavy / Purple / Hexagon) will test for six of the 24 possible pairs (Figure 8).
Figure 8 - Pairs
| Coverage of pairs after test 1 (6/24 = 25%) | |||||
|---|---|---|---|---|---|
| Large & Heavy | Small & Heavy | ||||
| Large & Light | Small & Light | ||||
| Large & Purple | Small & Purple | Heavy & Purple | Light & Purple | ||
| Large & Green | Small & Green | Heavy & Green | Light & Green | ||
| Large & Hexagon | Small & Hexagon | Heavy & Hexagon | Light & Hexagon | Purple & Hexagon | Green & Hexagon |
| Large & Circle | Small & Circle | Heavy & Circle | Light & Circle | Purple & Circle | Green & Circle |
So after the first Test case, the coverage chart will show that 25% of the total possible pairs that could be tested in this simple example have actually been tested at this point (Figure 9).
Figure 9 - Interactions
The second scenario (Small / Light / Purple / Circle) will test another six pairs. Importantly, none of these six pairs of values have been tested yet. In our first two Tests, we will have tested a total of 12 pairs of values.
Figure 10 - Pairs
So after two Test cases, the chart shows that 50% of the possible pairs (e.g., 12 tested out of 24 possible) have been tested (Figure 11).
Figure 11 - Interactions
Why do coverage charts start with a steep trajectory (with lots of added coverage per Test) only to flatten out towards the end (with only a little added coverage per Test)? Analyzing Test number 3 shows us why (Figure 12):
Figure 12 - Pairs
There is no way to select values that will test six new pairs, as we did in each of the first two Tests. The best we can do is test five new pairs and one previously tested pair.
In this third Test, "Large and Hexagon" has already been tested in the first Test. After Test 3, we have now tested 17 of the 24 total possible pairs.
The coverage chart shows 70.8% (versus 75% if we had been able to include six new pairs in Test 3; Figure 13).
Figure 13 - Interactions
What’s going on with the final two Test cases? We achieved 25% coverage of pairs in Test 1 and Test 2, so why do Test 5 and Test 6 only add a small 4.2% increase each?
The final two scenarios add only a tiny amount of coverage because we managed to test all but two pairs of values in the first four Test cases. It will take at least two more Test cases to cover those last two remaining pairs.
The only pair tested for the first time in Scenario 5 is "Small and Hexagon." The only new pair tested in Scenario 6 is "Large and Circle." This is just one-sixth as many new pairs in each test compared to the first two Tests.
Figure 14 - Pairs
The likelihood of finding a new defect in Test number 5 or 6 is much lower than finding a new defect in Test case 1 or 2.
To Consider When Analyzing Coverage Information
First, when used correctly and thoughtfully, the coverage information can be extremely useful. It provides a quick method for objectively assessing, "How much additional testing coverage am I achieving with each new Test?" and "How much testing is enough?". Many Testing teams use a rule of thumb, such as stopping the execution of Xray Test Case Designer-generated Tests after achieving 80% coverage, since they can clearly see diminishing returns to further testing beyond that point.
The second point to keep in mind is cautionary. It would be a mistake to look at the graph, see that 100% coverage has been achieved after the final Xray Test Case Designer-generated Test, and conclude that these Tests cover everything that should be tested. 100% of what coverage?
An analysis chart generated by the Xray Test Case Designer, like all software testing coverage reports, is an imperfect model of what should be covered (which itself is based on an imperfect model of the system under test). There may be significant aspects of the system that were not included in the Parameters screen. One or more of these excluded aspects - such as hardware configuration, software configuration or plug-ins, the order in which actions are executed, whether a user is navigating with a mouse or keyboard, or whether submit buttons are clicked multiple times quickly - could potentially cause defects that your current Test Set might not identify.