Overview
Learn how to use Analysis to understand specific interaction coverage after every scenario.
Coverage Matrix
Click on the “Analysis→ Coverage Matrix” menu link to use this feature:
Understanding the Matrix
Each square of the matrix chart represents a pair. How to read the chart:
- Read the values listed across the top
- Read the values listed down the left side
- Find an intersection of two values (one from top and one from side):
- If it is green: the pair is already covered.
- If it is red: the pair is not covered yet given the scenario you selected via slider.
- If it is black: the pair is marked as invalid.
Slide your way to precise coverage
The slider at the top allows you to visualize which pairs are covered with each test.
When you arrive at the Matrix Chart screen, you’ll automatically see the chart cycle from Test Case 0 through Test Case N (the final test in your set). By default, the slider will “settle” at the position that achieves around 80% coverage.
Communicate your testing coverage precisely
With the Matrix Chart, communicating your testing coverage has been made even easier! Imagine a situation where you have created some tests and your manager comes by and says, “Development is running behind, so we’ll only have time to execute 2/3s of your tests before Go-Live. What will be missed?” You can easily respond with a slide like the one below:
With this information in mind, you can then weigh the testing options. Do we need to test for this 2-way interaction which is still a gap by test 10? Is it worth generating 6 more test cases? Are we better off adding a forced interaction to bump that pair's priority? The Coverage Matrix can be invaluable in the decision-making process.
- Can I see 3-way and higher interactions?
At this time, the Matrix Chart only supports demonstrating the pairs that are covered with each test. - Special Note for Color-Blind Users
Pressing “C” on your keyboard changes the color scheme of the Matrix Chart. The following 3 options are available: Red/Green/Black (default), Orange/Blue/Black, Purple/Orange/Black. - Dealing with Large Number of Values
Having 50+ total values can appear to be overwhelming in the Matrix Chart. Xray Test Case Designer can help make such use cases more manageable with Zoom and Click-Drag features:- Each matrix cell has a pop-up window with the details in larger font (compared to value names).
- Scrolling mouse wheel while your cursors hovers over the matrix will increase or decrease the zoom of the section in the view.
- In the zoomed in state, you can left click the matrix, hold & drag to bring a different segment into visibility.
Coverage Graph
The key difference is that the Graph can be applied to any generated strength (3-way, etc.) but will only show the cumulative coverage percentage, not the specific interactions:
The shape of the curve is important to highlight: Xray Test Case Designer algorithm frontloads as much variation as possible, therefore the diminishing returns on incremental coverage improvement are reach fairly fast and the scenarios in the later part of the test suite do not contribute nearly as much as the first few.
Note: the order is based on the Index column from the Scenarios screen. Sorting the data table in a different way does not impact the Graph or Matrix as they always display the recommended execution priority. If you want to analyze the modified Scenarios table, you have to either freeze it or manually re-import it as Forced Interactions.
A few important points to consider when analyzing coverage information
- First, when used correctly (and thoughtfully), the information can be extremely useful. It gives you a quick method for objectively assessing "how much extra testing coverage am I achieving with each new test?" and "how much testing is enough?".
Many testing teams have a rule-of-thumb. For example, to stop executing the Xray Test Case Designer generated tests after they have achieved 80% coverage because they can clearly see diminishing marginal returns to further testing after that point. - The second thing to keep in mind is cautionary. As George Box says, "All models are wrong. Some models are useful." It would be a mistake to look at the graph, see that "100% coverage" has been achieved after the final Xray Test Case Designer generated test, and conclude that the tests cover everything that should be tested. It matters "100% of which coverage".
An "Analysis" chart generated by Xray Test Case Designer, like all software testing coverage reports, is an imperfect model of what should be covered (which is itself based on an imperfect model of the System Under Test). There could be significant aspects of the System Under Test that were not entered into the "Parameters" screen. It is important to remember that one or more of those excluded aspects (hardware configuration, software configuration or plug-in, the order in which actions are executed, whether a user is navigating with mouse or a keyboard, whether or not "submit" buttons are clicked multiple times quickly, etc.) could potentially cause a defect that might not be identified in your current set tests.