Status of a Test… for some scope
The “latest”/consolidated status of the Test for a specific scope/context
- Depends on where/how you’re analyzing your Test
- Generally it represents the latest result obtained for that Test in that context
- More grained or high-level overview
- e.g. status of a Test for version 1.0, depends on consolidating the latest results for Chrome and Firefox
- Can involve consolidating results from multiple Test Environments
Note: the TestRunStatus custom field provides a sneak peek for it (explained ahead).
Test Status in different scopes and Xray elements
Test Coverage panel in the “requirement” issue screen | “Tests” legacy report (on project tab) | Traceability Report |
---|---|---|
Calculating the status of a Test for a Scope
1 - Obtain Test Runs based on the analysis scope (i.e. the context)
2 - Obtain all the distinct environments involved in the previous Test Runs
3 - For each Test Environment, obtain the latest Test Run
if final statuses have precedence then
…. pick most recent Test Run based on its “finished on” date, having a status marked as “final”; if no T.R. exists, then the latest intermediate one (see following rule)
else,
pick most recent Test Run based on its “created” date
4 - If results from multiple environments are involved, then a joint status must be calculated:
- FAIL status wins over all other status
- Any status wins over PASS
- Based on their ranking, in the Test Statuses settings
Check your "Final statuses have precedence over non-final statuses” setting (default=true).
Analysis Scope – What is it?
The analysis scope is our context. Defines the criteria for finding out the relevant Test Runs.
- Version “V”
- Only Test Runs from Test Executions assigned to version V are considered
- Test Plan “T”
- Only Test Runs from Test Executions assigned to Test Plan T are considered
- Latest results / No version
- “Latest” Test Runs from ”recent” Test Executions are considered
And optionally…
- Test Environment “E”
- Only Test Runs from Test Executions assigned to Test Environment E are considered
If no Test Environment is specified, then Xray calculates the status based on the joint values for all the different Test Environments having scheduled executions
Lets see some examples....
Example 1 - "Final statuses have precedence over non-final statuses” setting as true
Example 2 - "Final statuses have precedence over non-final statuses” setting as false
Example 3 - Multiple Environments
When Final statuses have precedence over non-final statuses” setting as true.
When Final statuses have precedence over non-final statuses” setting as false.