Status of a Test… for some scope
The “latest”/consolidated status of the Test for a specific scope/context
- Depends on where/how you’re analyzing your Test on
- Generally, it represents the latest result obtained for that Test in that context
- More grained or high-level overview
- e.g. status of a Test for version 1.0, depends on consolidating the latest results for Chrome and Firefox
- Can involve consolidating results from multiple Test Environments
Note: the TestRunStatus custom field provides a sneak peek for it (explained ahead)
Test Status in different scopes and Xray elements
Test Coverage panel in the “requirement” issue screen | “Tests” legacy report (on project tab) | Traceability Report |
---|---|---|
Calculating the status of a Test for a Scope
1 - Obtain Test Runs based on the analysis scope (i.e. the context)
2 - Obtain all the distinct environments involved in the previous Test Runs
3 - For each Test Environment, obtain the latest Test Run
if final statuses have precedence then
…. pick most recent Test Run based on its “finished on” date, having a status marked as “final”; if no T.R. exists, then the latest intermediate one (see following rule)
else,
pick most recent Test Run based on its “created” date
4 - If results from multiple environments are involved, then a joint status must be calculated:
- FAIL status wins over all other status
- Any status wins over PASS
- Based on their ranking, in the Test Statuses settings
Check your "Final statuses have precedence over non-final statuses” setting (default=true)
Analysis Scope – What is it?
The analysis scope, it’s our context. Defines the criteria for finding out the relevant Test Runs.
- Version “V”
- Only Test Runs from Test Executions assigned to version V are considered
- Test Plan “T”
- Only Test Runs from Test Executions assigned to Test Plan T are considered
- Latest results / No version
- “Latest” Test Runs from ”recent” Test Executions are considered
And optionally…
- Test Environment “E”
- Only Test Runs from Test Executions assigned to Test Environment E are considered
If no Test Environment is specified, then Xray calculates the status based on the joint values for all the different Test Environments having scheduled executions