Page History

...

Thus, when speaking about the "status of a Test", we need to give it additional context (e.g. "In which version?") since it depends on "where" and how you want to analyze it.

Xray provides some built-in Test statuses (which can’t be modified nor deleted):

• TODO  – Test is pending execution; this is a non-final status;
• EXECUTING  – Test is being executed; this is a non-final status; at least one step is mapped to a non-final Test Run status 
• FAIL  – Test failed
• ABORTED  – Test was aborted
• PASS  – Test passed successfully
• Each of this status maps to a coverage status, accordingly with the following table.

The status (i.e., result) of a Test Run is an attribute of the Test Run (a “Test Run” is an instance of a Test and is not a Jira issue) and is the one taken into account to assess the coverage status of the coverable issue.

Managing Test Statuses

Creating new Test (Run) statuses may be done in the Global Settings: Test Statuses configuration section of Xray.

...

The status of a Test Step indicates the result obtained for that step for some Test Run. 

Xray provides some built-in Test Step statuses (which can’t be modified nor deleted).

Managing Test Step Statuses

...

Thus, when speaking about the "status of a requirement/Story", for example, we need to give it additional context (e.g., "In which version?") since it depends on "where" and how you want to analyze it.

In Xray, for a given coverable issue, considering the default settings, its coverage status may be:

• OK – issue has been successfully and fully validated; all the Tests associated with the issue are PASSED
• NOK – issue is unsuccessfully validated; at least one Test associated with the issue is FAILED
• NOTRUN – issue has not been validated completely; at least one Test associated with the issue  is TODO or ABORTED and there are no Tests with status FAILED
• UNKNOWN – issue is in unknown state; at least one Test associated with the issue is UNKNOWN and there are no Tests with status FAILED
• UNCOVERED – issue is not covered with tests; the issue has no Tests associated to it

It’s not possible to create custom Test Coverage statuses.

...

The following table provides some examples given the Test Step Statuses configuration shown above.

...

...

...

...

...

...

Calculation of the status for a given Test

It is possible to calculate the status of a Test either by Version or Test Plan, in a specific Test Environment or globally, taking into account the results obtained for all Test Environments.

Analysis:

• By Version: For a given Test X, in order to calculate the coverage status for version V, we need to evaluate related Test Runs that were executed in that same version V. A special case is when you don't have versions or simply don't want to calculate the status based on a version (i.e., "No Version")
• By Test Plan: For a given Test X, in order to calculate the coverage status for Test Plan TP, we need to evaluate the related Tests Runs that were executed on Test Executions associated with Test Plan TP.
• On a specific Test Environment: For a given Test X, if a specific Environment is also chosen, then only Test Runs from Test Executions with this Environment will be considered. In case no Environment is specified, then all Test Executions are considered (more info here).

What affects the calculation:

...

Calculate the status of some Test, in version V or Test Plan TP, for Test Environment TE

1. This takes into account Test Runs in version V (as a result of Test Executions in version V) or Test Runs in Test Plan TP (within Test Executions associated with Test Plan TP)
2. If Test Environment is chosen, then only Tests Runs on that Environment (e.g., TE) will be considered.
3. If "Final statuses have precedence over non-final statuses" is true, then:
   1. final Test Run statuses will have higher ranking than non-final ones
   2. only the latest Test Run is taken into account based on its "finished on" date  
4. If "Final statuses have precedence over non-final statuses" is false, then:
   1. only the latest Test Run is taken into account based on its "created" date (i.e. the creation date of the related Test Execution)

Calculate the status of some Test, in version V or Test Plan TP, for "All Environments"

1. calculate the Test status for each Test Environment, based on all the implicit Test Environments from the relevant Test Executions (i.e., Test Executions in version V or Test Executions associated with Test Plan TP)
2. calculate the joint value for the Test status
   1. PASS has lowest ranking (i.e. for the calculated to be PASS, all calculated statuses must be PASS in the different Test Environments)
   2. if one is FAIL, then the calculated value will be FAIL
   3. otherwise, use the ranking of Test statuses

Examples

The following table provides some examples given the Test Statuses configuration shown above in the Managing Test Statuses section.

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

Calculation of the coverage status for a given issue

It is possible to calculate the test coverage status of a coverable issue either by Version or Test Plan, in a specific Test Environment or globally, taking into account the results obtained for all Test Environments.

Analysis :

• By Version: For a given issue X, in order to calculate the coverage status for version V, we need to evaluate the related Tests statuses that were executed on that same version V.
• By Test Plan: For a given issue X, in order to calculate the coverage status for Test Plan TP, we need to evaluate the related Tests statuses that were executed on Test Executions associated with Test Plan TP.
• On a specific Test Environment: For a given Test X, if a specific Environment is also chosen, then only Test Runs from Test Executions with this Environment will be considered. In case no Environment is specified, then all Test Executions are considered (more info on Test Environments here).

The algorithm is similar to the overall calculation of the Test status, taking into account the results obtained for different Test Environments.

...

The Tests that will be considered as covering the issue are not just the ones directly linked to the issue. In fact, they may either be direct ones or ones linked to "child" issues (e.g., sub-requirements). 

Algorithm :

1. Obtain the list of Tests that directly or indirectly through "child" issues (e.g., sub-requirements) cover the issue
   1. This depends on the Test Coverage Hierarchy-related settings, defined in Project Settings: Test Coverage
2. Calculate the Test status for all the Tests individually, in version V or Test Plan TP
   1. This takes into account Test Runs in version V (as a result of Test Executions in version V) or Test Runs in Test Plan TP (within Test Executions associated with Test Plan TP)
   2. If a specific Environment is also chosen, then only Test Runs from Test Executions with this Environment will be considered. In case no Environment is specified then all Test Executions are considered (more info on Test Environments here).
3. Calculate the "joint" status of all the previous Test statuses (i.e., by comparing together each Test status)
4. Calculate the coverage status mapped to the previous calculated Test status

What affects the calculation:

...

The calculation follows the rules described in the following table.

From another perspective, you would obtain the same value for the calculation of the status of the parent issue if you consider that it is being covered by all the explicitly linked Tests and also the ones linked to the child issues.

...

The following table provides some examples given the Test Statuses configuration shown above in the Managing Test Statuses section.

...

...

...

...

...

...

...

Setup information for possible use cases

1. I want to skip some Tests and proceed as they didn't exist
   1. Create a "Test Step Status"  (e.g., "SKIP"), mapped to the Test Status "PASS"
2. I want to fail a Test Run but I don't want to mark the requirement as being NOT OK because this failure can be discarded
   1. Create a "Test Status" (e.g., "FAIL_DISCARD") , non-final and mapped to the coverage status "UNKNOWN"; setting the status as non-final will give priority to other Test Runs you may have for that Test, If “Final Statuses have precedence over non-final” flag is enabled
   2. Create a "Test Step Status" (e.g., "IRRELEVANT_FAIL") and map it to the Test Status created in the previous step
3. I want to always see, for a given Test, the status of Test based on the last run scheduled for it, no matter if it was completed (i.e. in a final status) or not
   1. Just uncheck the flag “Final Statuses have precedence over non-final”
4. I want to execute some steps, set them as failed or passed, but I don't want them to reflect immediately in the status of the Test Run
   1. Create custom, non-final, Test statuses for passing and failure (e.g., "MYPASS", "MYFAIL"), mapped to the OK and NOK coverage statuses, respectively
   2. Create your own custom Test Step statuses for passed and failure (e.g., "PASS_CONTINUE" and FAIL_CONTINUE"), mapped to the previously created Test statuses