Status of a Test… for some scope

The “latest”/consolidated status of the Test for a specific scope/context

  • Depends on where/how you’re analyzing your Test 
  • Generally it represents the latest result obtained for that Test in that context
  • More grained or high-level overview
    • e.g. status of a Test for version 1.0, depends on consolidating the latest results for Chrome and Firefox
    • Can involve consolidating results from multiple Test Environments



Note: the TestRunStatus custom field provides a sneak peek for it (explained ahead). 

Test Status in different scopes and Xray elements


Test Coverage panel in the “requirement” issue screen

Indirectly in Overall Test Results

Traceability Report

Calculating the status of a Test for a Scope

1 - Obtain Test Runs based on the analysis scope (i.e. the context)​

2 - Obtain all the distinct environments involved in the previous Test Runs​

3 - For each Test Environment, obtain the latest Test Run​

if final statuses have precedence then​

…. pick most recent Test Run based on its “finished on” date, having a status marked as “final”; if no T.R. exists, then the latest intermediate one (see following rule)​

 else, ​

pick most recent Test Run based on its “created” date​

​

4 - If results from multiple environments are involved, then a joint status must be calculated:​

  1. FAIL status wins over all other status​
  2. Any status wins over PASS ​
  3. Based on their ranking, in the Test Statuses settings

​

Check your "Final statuses have precedence over non-final statuses” setting (default=true). 




Analysis Scope – What is it?​


The analysis scope is our context. Defines the criteria for finding out the relevant Test Runs.​

​

  • Version “V”​
  • Only Test Runs from Test Executions assigned to version V are considered​
  • Test Plan “T”​
  • Only Test Runs from Test Executions assigned to Test Plan T are considered​
  • Latest results / No version​
  • “Latest” Test Runs from ”recent” Test Executions are considered​

​

And optionally… ​

​

  • Test Environment “E”​
  • Only Test Runs from Test Executions assigned to Test Environment E are considered​

​

If no Test Environment is specified, then Xray calculates the status based on the joint values for all the different Test Environments having scheduled executions​


Lets see some examples....

Example 1  - ​"Final statuses have precedence over non-final statuses” setting as true



​

Example 2  - "Final statuses have precedence over non-final statuses” setting as false


Example 3 - Multiple Environments

When Final statuses have precedence over non-final statuses”  setting as true​. 



When Final statuses have precedence over non-final statuses”  setting as false​.