...

A coverable issue (e.g., Story, requirement) may be either covered by one or multiple Tests. In fact, the test coverage status of a given issue goes way further than the basic covered/not covered information: it will take ; it takes into account your test results.

As soon as you start running your Tests, the individual test result may be one of many and it will be very specific to your use case.

...

Requirements may be validated directly or indirectly , through the related sub-requirements and associated Test cases.

Thus, how How do all these factors contribute to the calculation of a coverage status? How is evaluated the status of some a Test evaluated?

Let’s start by detailing the different possible values Test, Test Step and coverage statuses. In the endThen, we’ll see how they’ll impact on the calculation of the coverage status of a issue in some specific version or Test Plan.

Overview of statuses

Whenever When talking about statuses, we may be talking about statuses of requirements, Tests, Test Runs and Test Steps.

The status (i.e., test coverage status) of a coverble coverable issue depends on the status of the its "related" Tests.

The status of a Test depends on the status of the its "related" Test Runs, which in turn depend on the recorded Test Step statuses for each one of them.

...

The status of a Test tells you information about its current consolidated state (e.g. latest record result, if existent). Was it is executed? Successfully? In which version?

Thus, whenever when speaking about the "status of a Test", we need to give it some additional context (e.g. "In which version?") since it depends on "where" and how you want to analyze it.

...

The status of a Test indicates its "latest state" in some given context (e.g. for some version, some Test Plan and/or in some Test Environment).

Xray provides some built-in Test statuses (which can’t be modified nor deleted):

TODO  – Test is pending execution; this is a non-final status;

EXECUTING  – Test is being executed; this is a non-final status; at least one step is mapped to a non-final Test Run status 

FAIL  – Test failed

ABORTED  – Test was aborted

PASS  – Test passed successfully

Each of this status maps to a coverage status, accordingly with the following table.

Test status

Final status?

Test Coverage Status mapped to

PASS

yes

OK

FAIL

yes

NOK

TODO

no

NOTRUN

ABORTED

yes

NOTRUN

EXECUTING

no

NOTRUN

custom

custom

OK, NOK, NOTRUN or UNKNOWN

The status (i.e., result) of a Test Run is an attribute of the Test Run (a “Test Run” is an instance of a Test and is not a Jira issue) and is the one taken into account to assess the coverage status of the coverable issue.

...

Creating new Test (Run) statuses may be done in the Global Settings: Test Statuses configuration section of Xray.

Whenever When creating/editing a Test status, we have to identify the Test Coverage Status to which we want this Test status to map to.

One important attribute of a Test status is the “final” attribute. If “Final Final Statuses have precedence over non-final” final flag is enabled, then Xray will give priority to final statuses whenever when calculating the status of a Test. In other words, if you have a Test currently in some a final status (e.g., PASS, FAIL) and you schedule a new Test Run for it, then this Test Run won't affect the calculation of the status of the Test.

This may be used, when users You may use this if you prefer to take into account only the last final/complete recorded result and want to discard Test Runs that are in an intermediate status (e.g., EXECUTING, TODO).

Test Step

...

status

The status of a Test Step indicates the result obtained for that step for some Test Run. 

...

Statuses reported at Test Step level will contribute to the overall calculation of the status of the related Test Run.

Xray provides some built-in Test Step statuses (which can’t be modified nor deleted).

Test Step status

Test status

PASS

PASS

TODO

TODO

EXECUTING

EXECUTING

FAIL

FAIL

custom

custom

Managing Test Step Statuses

Creating new Test Step statuses may be done in the Global Settings: Test Step Statuses configuration section of Xray.

Whenever When creating/editing a Test Step status, we have to identify the Test status to which we want this step status to map to.

Note that native Test Step statuses can’t be modified nor deleted.

...

The (coverage) status of a coverable issue tells you information about its current state, from a quality perspective. Is it covered with test cases? If so, has it been validated successfully? In which version?

Thus, whenever when speaking about the "status of a requirement/Story", for example, we need to give it some additional context (e.g., "In which version?") since it depends on "where" and how you want to analyze it.

...

The (coverage) status of a coverable issue indicates its coverage information along with its "state", depending on the results recorded for the Tests that do validate it.

This status is evaluated in

...

a given context (e.g., for some version, some Test Plan and/or in some Test Environment).

In Xray, for a given coverable issue, considering the default settings, its coverage status may be:

OK – issue has been successfully and fully validated; all the Tests associated with the issue are PASSED

NOK – issue is unsuccessfully validated; at least one Test associated with the issue is FAILED

NOTRUN – issue has not been validated completely; at least one Test associated with the issue  is TODO or ABORTED and there are no Tests with status FAILED

UNKNOWN – issue is in unknown state; at least one Test associated with the issue is UNKNOWN and there are no Tests with status FAILED

UNCOVERED – issue is not covered with tests; the issue has no Tests associated to it

It’s not possible to create custom Test Coverage statuses.

You can see that in order to To calculate an issue coverage status , for some a specific system version, we “just” need to take into account the status of the related Tests for that same version. We’ll come back to this later on.

Calculation of the status for a given Test Run

The status of a given Test Run is an attribute that most of the times that is often calculated automatically on automatically based on the respective recorded step statuses. User You can also enforce a specific status for a Test Run, which in turn may implicitly enforce specific step statuses (e.g., setting a Test Run as "FAIL" can set all steps as "FAIL"). 

This calculation is made by comparing steps together, following these rules:

• if a step status is mapped to a non-final, then Test Run status will be "EXECUTING"
• compare all the step statuses

...

• based on their order (steps at the bottom of the list will have higher ranking)
• the step status "PASS" has the lowest ranking 

The order of the steps is indifferent irrelevant for the purpose of the overall Test Run status value.

Examples

The following table provides some examples given the Test Step Statuses configuration shown above.

Example #

Statuses of the steps/contexts

(the order of the steps/contexts is irrelevant)

Calculated value for the status of the Test Run

Why?

1

PASS

PASS

PASS

PASS

All steps are PASS; thus,

...

the joint value is PASS

2

PASS

TODO

PASS

EXECUTING

At least one step status (i.e., TODO) is mapped to a non-final Test status

3

PASS

FAIL

PASS

FAIL

One of the step statuses (i.e., FAIL) has higher ranking than the other ones

4

XPASS

FAIL

PASS

FAIL

XPASS has higher ranking than the other ones

...

; thus the overall calculated value is based on the mapping for that Test Step status.

5

FAIL

XPASS

FAIL

FAIL

XPASS has higher ranking than the other ones

...

; thus the overall calculated value is based on the mapping for that Test Step status.

Calculation of the status for a given Test

It is possible to calculate the status of a Test either by Version or Test Plan, in a specific Test Environment or globally, taking into account the results obtained for all Test Environments.

Analysis:

By Version: For a given Test X, in order to calculate the coverage status for version V, we need to evaluate

...

related

...

Test Runs that were executed

...

in that same version V. A special case is

...

when you don't have versions or simply don't want to calculate the status based on a version (i.e., "No Version")

By Test Plan: For a given Test X, in order to calculate the coverage status for Test Plan TP, we need to evaluate the related Tests Runs that were executed on Test Executions associated with Test Plan TP.

On a specific Test Environment: For a given Test X, if a specific Environment is also chosen, then only Test Runs from Test Executions with this Environment will be considered. In case no Environment is specified, then all Test Executions are considered (more

...

info here).

What affects the calculation:

• the flag "Final statuses have precedence over non-final statuses" shown in the calculation sections (enabled by default)
• the existence of Test Runs for different Test Environments, in case the analysis is made for "All Environments"

Calculate the status of some Test, in version V or Test Plan TP, for Test Environment TE

This takes into account Test Runs in version V (as a result of Test Executions in version V) or Test Runs in Test Plan TP (within Test Executions associated with Test Plan TP)

If Test Environment is chosen, then only Tests Runs on that Environment (e.g., TE) will be considered.

If "Final statuses have precedence over non-final statuses" is true, then:

• final Test Run statuses will have higher ranking than non-final ones
• only the latest Test Run is taken into account based on its "finished on" date  

If "Final statuses have precedence over non-final statuses" is false, then:

• only the latest Test Run is taken into account based on its "created" date (i.e. the creation date of the related Test Execution)

Calculate the status of some Test, in version V or Test Plan TP, for "All Environments"

calculate the Test status for each Test Environment, based on all the implicit Test Environments from the relevant Test Executions (i.e.

...

, Test Executions in version V or Test Executions associated with Test Plan TP)

calculate the joint value for the Test status

PASS has lowest ranking (i.e. for the calculated to be PASS, all calculated statuses must be PASS in the different Test Environments)

if one is FAIL, then the calculated value will be FAIL

otherwise, use the ranking of Test statuses

Examples

The following table provides some examples given the Test Statuses configuration shown above in the Managing Test Statuses section.

Example #

Statuses of the Test Runs

(ordered by time of execution/creation, ascending)

Final statuses have precedence over non-final statuses

Calculated value for the status of the Test

Why?

1a

PASS

PASS

TODO

true

PASS

Latest executed Test Run (2) having a final status was PASS.

1b

PASS

PASS

TODO

false

TODO

Latest created Test Run (3) was TODO.

2a

PASS (env1)

MYPASS2 (env2)

TODO (env2)

PASS (env3)

true

XPASS

Latest executed final Test Runs on each environment were PASS, MYPASS2 and PASS, respectively.

Since MYPASS2 (3) has a higher ranking

...

, the calculated status will be MYPASS2.

2b

PASS (env1)

MYPASS2 (env2)

TODO (env2)

PASS (env3)

false

XPASS

Latest created Test Runs on each environment were PASS, TODO and PASS, respectively.

Since PASS has the lowest ranking,

...

TODO (3) will "win" and then the calculated status will be TODO

3

PASS (env1)

TODO (env2)

PASS (env3)

true

TODO

Latest created Test Runs on each environment were PASS, TODO and PASS, respectively.

Although Test Environment "env2" has only a non-final Test Run, since there is no other Run for that environment, then it will be considered as the calculated status for that environment.

Since PASS has the lowest ranking, then TODO (3) will "win" and then the calculated status will be TODO.

4

PASS (env1)

FAIL (env2)

PASS (env3)

true (or false)

FAIL

Latest executed (or created) final Test Runs on each environment were PASS, FAIL and PASS, respectively.

Since the calculated status for one of the environments is FAIL,

...

the calculated status will be FAIL.

5

PASS (env1)

MYPASS2 (env2)

TODO (env2)

MYFAIL (env3)

true

MYPASS2

Latest executed final Test Runs on each environment were PASS, MYPASS2 and MYFAIL, respectively.

MYPASS2 has a higher ranking than the other ones; thus,

...

the overall calculated value will be MYPASS2.

6

PASS (env1)

MYPASS2 (env2)

TODO (env2)

MYFAIL (env3)

false

MYFAIL

Latest created Test Runs on each environment were PASS, TODO and MYFAIL, respectively.

MYFAIL has a higher ranking than the other ones; thus,

...

the overall calculated value will be MYFAIL.

Calculation of the coverage status for a given issue

It is possible to calculate the test coverage status of a coverable issue either by Version or Test Plan, in a specific Test Environment or globally, taking into account the results obtained for all Test Environments.

Analysis :

By Version: For a given issue X, in order to calculate the coverage status for version V, we need to evaluate the related Tests statuses that were executed on that same version V.

By Test Plan: For a given issue X, in order to calculate the coverage status for Test Plan TP, we need to evaluate the related Tests statuses that were executed on Test Executions associated with Test Plan TP.

On a specific Test Environment: For a given Test X, if a specific Environment is also chosen, then only Test Runs from Test Executions with this Environment will be considered. In case no Environment is specified, then all Test Executions are considered (more info on Test Environments here).

The algorithm is similar to the overall calculation of the Test status, taking into account the results obtained for different Test Environments.

In other words, the status for each linked and "relevant" Test case is calculated and in at the end, a joint calculation is done for a virtual Test case. The coverage status will correspond to the mapped value for the status that was calculated for this virtual Test.

The Tests that will be considered as covering the issue are not just the ones directly linked to the issue. In fact, they may either be direct ones or ones linked to "child" issues (e.g., sub-requirements). 

Algorithm :

Obtain the list of Tests that directly or indirectly through "child" issues (e.g., sub-requirements) cover the issue

This depends on the Test Coverage Hierarchy-related settings, defined in Project Settings: Test Coverage

...

Calculate the Test status for all the Tests individually, in version V or Test Plan TP

This takes into account Test Runs in version V (as a result of Test Executions in version V) or Test Runs in Test Plan TP (within Test Executions associated with Test Plan TP)

If a specific Environment is also chosen, then only Test Runs from Test Executions with this Environment will be considered. In case no Environment is specified then all Test Executions are considered (more info on Test Environments here).

Calculate the "joint" status of all the previous Test statuses (i.e., by comparing together each Test status)

Calculate the coverage status mapped to the previous calculated Test status

What affects the calculation:

...

Indirectly, the flag "Final statuses have precedence over non-final statuses" (enabled by default)

...

The existence of Test Runs for different Test Environments, in case the analysis is made for "All Environments"

Test Coverage Hierarchy

Sometimes, you may have parent requirements and sub-requirements, for example. In general, you may have parent issues and "child" issues, both of them being handled as coverable issues.

Xray provides is able to understand this hierarchical relation and takes that into account for the calculation of the coverage status of the parent issues.

...

The calculation follows the rules described in the following table.

PARENT \ CHILD

OK

NOK

NOT RUN

UNKNOWN

UNCOVERED

OK

OK

NOK

NOT RUN

UNKNOWN

OK

NOK

NOK

NOK

NOK

NOK

NOK

NOT RUN

NOT RUN

NOK

NOT RUN

UNKNOWN

NOT RUN

UNKNOWN

UNKNOWN

NOK

UNKNOWN

UNKNOWN

UNKNOWN

UNCOVERED

OK

NOK

NOT RUN

UNKNOWN

UNCOVERED

From another perspective, you would obtain the same value for the calculation of the status of the parent issue if you consider that it is being covered by all the explicitly linked Tests and also the ones linked to the child issues.

Consequences :

...

The parent issue is OK if it is NOK per

...

se and the child issues are either UNCOVERED or also OK

...

The parent issue is NOK if it is NOK per

...

se or if any of the child issues is NOK

...

The parent issue is only UNCOVERED if neither the parent requirement is covered per

...

se nor the child issues are covered

...

...

Even if you are using the Test Coverage Hierarchy-related features, when you have tests that are directly linked to the parent issue, Xray assumes that you are validating the parent issue directly. Thus, it's irrelevant if the child issues are uncovered by tests or not.

Examples

The following table provides some examples given the Test Statuses configuration shown above in the Managing Test Statuses section.

Example #

Statuses of the related Tests

(child issues, whenever present, appear as subReqX)

Calculated value for the coverage status of the issue

Why?

1

PASS

PASS

PASS

OK

All Tests are passed (it is similar to having just one virtual test that would be considered PASS and thus mapped to the OK status of the issue)

2

PASS

PASS

TODO

NOT RUN

One of the Tests (3) is TODO, which has a higher ranking than PASS.

3

PASS

PASS

FAIL

NOK

One of the Tests (3) is FAIL, which has a higher ranking than PASS.

4

PASS

subReq1 => OK

PASS

subReq2 => NOK

PASS

FAIL

NOK

One of the Tests (3b) is FAIL, thus subReq2 will be considered as NOK. Since it is NOK, then the parent req which has higher ranking than PASS.

5

PASS

subReq1 => NOTRUN

TODO

subReq2 => OK

PASS

PASS

NOT RUN

One of the child issues (subReq1) is NOT RUN; thus,

...

the calculated status,

...

when doing

...

in conjunction with the parent issue status

...

will be NOT RUN.

6

PASS

subReq1 => UNCOVERED

(no Tests associated)

subReq2 => UNCOVERED

(no Tests associated)

OK

Since all child issues are uncovered and the parent issue is covered directly by one Test (1), which is currently PASS, then the calculated

...

OK

...

status will be based on that Test.

Setup information for

...

possible use cases

I want to skip some Tests and proceed as they didn't exist

...

Create a "Test Step Status"  (e.g., "SKIP"), mapped to the Test Status "PASS"

I want to fail a Test Run but I don't want to mark the requirement as being NOT OK because this failure can be discarded

...

Create a "Test Status" (e.g., "FAIL_DISCARD") , non-final and mapped to the coverage status "UNKNOWN"; setting the status as non-final will give priority to other Test Runs you may have for that Test, If “Final Statuses have precedence over non-final” flag is enabled

...

Create a "Test Step Status" (e.g., "IRRELEVANT_FAIL") and map it to the Test Status created in the previous step

I want to always see, for a given Test, the status of Test based on the last run scheduled for it, no matter if it was completed (i.e. in a final status) or not

...

Just uncheck the flag “Final Statuses have precedence over non-final”

I want to execute some steps, set them as failed or passed, but I don't want them to reflect immediately in the status of the Test Run

...

Create custom, non-final, Test statuses for passing and failure (e.g., "MYPASS", "MYFAIL"), mapped to the OK and NOK coverage statuses, respectively

...

Create your own custom Test Step statuses for passed and failure (e.g., "PASS_CONTINUE" and FAIL_CONTINUE"), mapped to the previously created Test statuses