Versions Compared

Old Version 12

changes.mady.by.user Helder Ferreira

Saved on

compared with

New Version 13

changes.mady.by.user Sofia Canario

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The shape of the curve is important to highlight: Xray Test Case Designer algorithm frontloads as much variation as possible, therefore the diminishing returns on incremental coverage improvement are reach fairly fast and the scenarios in the later part of the test suite do not contribute nearly as much as the first few.

Note: the order is based on the Index column from the Scenarios screen. Sorting the data table in a different way does not impact the Graph or Matrix as they always display the recommended execution priority. If you want to analyze the modified Scenarios table, you have to either freeze it or manually re-import it as Forced Interactions.


Info

A few important points to consider when analyzing coverage information

  1. First, when used correctly (and thoughtfully), the information can be extremely useful. It gives you a quick method for objectively assessing "how much extra testing coverage am I achieving with each new test?" and "how much testing is enough?".
    Many testing teams have a rule-of-thumb. For example, to stop executing the Xray Test Case Designer generated tests after they have achieved 80% coverage because they can clearly see diminishing marginal returns to further testing after that point.

  2. The second thing to keep in mind is cautionary. As George Box says, "All models are wrong. Some models are useful." It would be a mistake to look at the graph, see that "100% coverage" has been achieved after the final Xray Test Case Designer generated test, and conclude that the tests cover everything that should be tested. It matters "100% of which coverage".
    An "Analysis" chart generated by Xray Test Case Designer, like all software testing coverage reports, is an imperfect model of what should be covered (which is itself based on an imperfect model of the System Under Test). There could be significant aspects of the System Under Test that were not entered into the "Parameters" screen. It is important to remember that one or more of those excluded aspects (hardware configuration, software configuration or plug-in, the order in which actions are executed, whether a user is navigating with mouse or a keyboard, whether or not "submit" buttons are clicked multiple times quickly, etc.) could potentially cause a defect that might not be identified in your current set tests.