Page History
...
| Info | ||
|---|---|---|
| ||
Whenever we're speaking about the status associated with a requirement or with a Test (and even with a Test Set), we may be talking about different things:
The current page details the first one, i.e. the status of the entities based on the executions made in some context. The "Requirement Status", "TestRunStatus" and "Test Set Status" custom fields are calculated fields, usable in the context of requirement, Test and Test Set issues, respectively, that calculate the "status" (not the workflow status) of the requirement/Test/Test Set for a specific version, considering the executions (i.e., test runs) for that version. The version for which it calculates the status depends on the behavior defined in a global configuration (see configuration details here). These fields will show the calculated status for the respective entity, for some specific version; they're just used to have a quick glimpse of the status of each entity, for example right in the view issue screen; therefore their usage is limited. More information on these and other custom fields can be found here . |
Test status
The status of a Test tells you information about its current consolidated state (e.g. latest record result, if existent). Was it is executed? Successfully? In which version?
Thus, whenever speaking about the "status of a Test" we need to give it some additional context (e.g. "In which version?") since it depends on "where" and how you want to analyze it.
| Info |
|---|
The status of a Test indicates its "latest state" in some given context (e.g. for some version, some Test Plan and/or in some Test Environment). |
Xray provides some built-in Test statuses (which can’t be modified nor deleted):
- TODO – Test is pending execution; this is a non-final status;
- EXECUTING – Test is being executed; this is a non-final status; at least one step is mapped to a non-final Test Run status
- FAIL – Test failed
- ABORTED – Test was aborted
- PASS – Test passed successfully
Each of this status maps to a requirement status, accordingly with the following table.
...
The status (i.e. result) of a Test Run is an attribute of the Test Run (a “Test Run” is an instance of a Test and is not a Jira issue) and is the one taken into account to assess the status of the requirement.
...
| title | Please note |
|---|
Do not mix up the status of a given test with the "TestRunStatus" custom field, which shows the status of a given Tests for a specific version, depending on the configuration under Custom Field Preferences. More info on this custom field here.
Test status
The status of a Test tells you information about its current consolidated state (e.g. latest record result, if existent). Was it is executed? Successfully? In which version?
Thus, whenever speaking about the "status of a Test" we need to give it some additional context (e.g. "In which version?") since it depends on "where" and how you want to analyze it.
| Info |
|---|
The status of a Test indicates its "latest state" in some given context (e.g. for some version, some Test Plan and/or in some Test Environment). |
Xray provides some built-in Test statuses (which can’t be modified nor deleted):
- TODO – Test is pending execution; this is a non-final status;
- EXECUTING – Test is being executed; this is a non-final status; at least one step is mapped to a non-final Test Run status
- FAIL – Test failed
- ABORTED – Test was aborted
- PASS – Test passed successfully
Each of this status maps to a requirement status, accordingly with the following table.
| Test status | Final status? | Requirement status mapped to |
|---|---|---|
| PASS | yes | OK |
| FAIL | yes | NOK |
| TODO | no | NOTRUN |
| ABORTED | yes | NOTRUN |
| EXECUTING | no | NOTRUN |
| custom | custom | OK, NOK, NOTRUN or UNKNOWN |
The status (i.e. result) of a Test Run is an attribute of the Test Run (a “Test Run” is an instance of a Test and is not a Jira issue) and is the one taken into account to assess the status of the requirement
...
.
Managing Test Statuses
Creating new Test (Run) statuses may be done in the Manage Test Statuses configuration section of Xray.
Whenever creating/editing a Test status, we have to identify the Requirement status we want this Test status to map to.
One important attribute of a Test status is the “final” attribute. If “Final Statuses have precedence over non-final” flag is enabled, then Xray will give priority to final statuses whenever calculating the status of a Test. In other words, if you have a Test currently in some final status (e.g. PASS, FAIL) and you schedule a new Test Run for it, then this Test Run won't affect the calculation of the status of the Test.
...
You can see that in order to calculate a requirement coverage status, for some specific system version, we “just” need to take into account the status of the related Tests for that same version. We’ll come back to this later on.
...
| title | Please note |
|---|
...
for that same version. We’ll come back to this later on.
Calculation of the status for a given Test Run
...
What affects the calculation:
...
- the separation of concerns setting, in Xray administration settings, in the tab "Requirement Coverage" (more info here)
- this affects how the status of a Test Run contributes to the status of each linked requirement; if disabled, users can enforce a specific mapping for each requirement, otherwise the Test Run status will contribute in the same way to all the linked requirements
- indirectly, the setting "Final statuses have precedence over non-final statuses" in Xray administration settings, in the tab "Test Statuses" (enabled by default)
- the existence of Test Runs for different Test Environments, in case the analysis is made for "All Environments"
...