Page History
...
- TODO – Test is pending execution; this is a non-final status;
- EXECUTING – Test is being executed; this is a non-final status; at least one step is mapped to a non-final Test Run status
- FAIL – Test failed
- ABORTED – Test was aborted
- PASS – Test passed successfully
- Each of this status maps to a coverage status, accordingly with the following table.
Test status | Final status? | Test Coverage Status mapped to |
|---|---|---|
| PASS | yes | OK |
| FAIL | yes | NOK |
| TODO | no | NOTRUN |
| ABORTED | yes | NOTRUN |
| EXECUTING | no | NOTRUN |
| custom | custom | OK, NOK, NOTRUN or UNKNOWN |
The status (i.e., result) of a Test Run is an attribute of the Test Run (a “Test Run” is an instance of a Test and is not a Jira issue) and is the one taken into account to assess the coverage status of the coverable issue.
...
Xray provides some built-in Test Step statuses (which can’t be modified nor deleted).
Test Step status | Test status |
|---|---|
| PASS | PASS |
| TODO | TODO |
| EXECUTING | EXECUTING |
| FAIL | FAIL |
| custom | custom |
Managing Test Step Statuses
...
The following table provides some examples given the Test Step Statuses configuration shown above.
Example # | Statuses of the steps/contexts (the order of the steps/contexts is irrelevant) | Calculated value for the status of the Test Run | Why? |
|---|---|---|---|
| 1 |
| PASS | All steps are PASS, thus the joint value is PASS |
| 2 |
| EXECUTING | At least one step status (i.e. TODO) is mapped to a non-final Test status |
| 3 |
| FAIL | One of the step statuses (i.e. FAIL) has higher ranking than the other ones |
| 4 |
| FAIL | XPASS has higher ranking than the other ones, thus the overall calculated value is based on the mapping for that Test Step status. |
| 5 |
| FAIL | XPASS has higher ranking than the other ones, thus the overall calculated value is based on the mapping for that Test Step status. |
Calculation of the status for a given Test
...
The following table provides some examples given the Test Statuses configuration shown above in the Managing Test Statuses section.
Example # | Statuses of the Test Runs (ordered by time of execution/creation, ascending) | Final statuses have precedence over non-final statuses | Calculated value for the status of the Test | Why? |
|---|---|---|---|---|
| 1a |
| true | PASS | Latest executed Test Run (2) having a final status was PASS. |
| 1b |
| false | TODO | Latest created Test Run (3) was TODO. |
| 2a |
| true |
| MYPASS2 | Latest executed final Test Runs on each environment were PASS, MYPASS2 and PASS respectively. Since MYPASS2 (3) has higher ranking then the calculated status will be MYPASS2. | |
| 2b |
| false |
|---|
| TODO | Latest created Test Runs on each environment were PASS, TODO and PASS respectively. Since PASS has the lowest ranking, then TODO (3) will "win" and then the calculated status will be TODO | |||
| 3 |
| true | TODO | Latest created Test Runs on each environment were PASS, TODO and PASS respectively. Although Test Environment "env2" has only a non-final Test Run, since there is no other Run for that environment, then it will be considered as the calculated status for that environment. Since PASS has the lowest ranking, then TODO (3) will "win" and then the calculated status will be TODO. |
|---|---|---|---|---|
| 4 |
| true (or false) | FAIL | Latest executed (or created) final Test Runs on each environment were PASS, FAIL and PASS respectively. Since the calculated status for one of the environments is FAIL, then the calculated status will be FAIL. |
| 5 |
| true | MYPASS2 | Latest executed final Test Runs on each environment were PASS, MYPASS2 and MYFAIL respectively. MYPASS2 has higher ranking than the other ones, thus the overall calculated value will be MYPASS2. |
| 6 |
| false | MYFAIL | Latest created Test Runs on each environment were PASS, TODO and MYFAIL respectively. MYFAIL has higher ranking than the other ones, thus the overall calculated value will be MYFAIL. |
Calculation of the coverage status for a given issue
...
The calculation follows the rules described in the following table.
PARENT \ CHILD | OK | NOK | NOT RUN | UNKNOWN | UNCOVERED |
|---|---|---|---|---|---|
| OK | OK | NOK | NOT RUN | UNKNOWN | OK |
| NOK | NOK | NOK | NOK | NOK | NOK |
| NOT RUN | NOT RUN | NOK | NOT RUN | UNKNOWN | NOT RUN |
| UNKNOWN | UNKNOWN | NOK | UNKNOWN | UNKNOWN | UNKNOWN |
| UNCOVERED | OK | NOK | NOT RUN | UNKNOWN | UNCOVERED |
From another perspective, you would obtain the same value for the calculation of the status of the parent issue if you consider that it is being covered by all the explicitly linked Tests and also the ones linked to the child issues.
...
The following table provides some examples given the Test Statuses configuration shown above in the Managing Test Statuses section.
Example # | Statuses of the related Tests (child issues, whenever present, appear as subReqX) | Calculated value for the coverage status of the issue | Why? |
|---|---|---|---|
| 1 |
| OK | All Tests are passed (it is similar to having just one virtual test that would be considered PASS and thus mapped to the OK status of the issue) |
| 2 |
| NOT RUN | One of the Tests (3) is TODO, which has higher ranking than PASS. |
| 3 |
| NOK | One of the Tests (3) is FAIL, which has higher ranking than PASS. |
| 4 |
| NOK | One of the Tests (3b) is FAIL, thus subReq2 will be considered as NOK. Since it is NOK, then the parent req which has higher ranking than PASS. |
| 5 |
| NOT RUN | One of the child issues (subReq1) is NOT RUN, thus the calculated status, whenever doing the conjunction with the parent issue status, will be NOT RUN. |
| 6 |
| OK | Since all child issues are uncovered and the parent issue is covered directly by one Test (1), which is currently PASS, then the calculated "OK" status will be based on that Test. |
Setup information for possible use cases
...