Overview
The “latest”/consolidated status of the coverable issue (e.g. requirement) for a specific context
- Depends on where/how you’re analyzing your ”requirement"
- Generally it represents the latest result obtained for that Test in that context
- More grained or high-level overview
- Can involve consolidating results from multiple Test Environments
- e.g. status of a Test for version 1.0, depends on consolidating the latest results for Chrome and Firefox
the “Requirement Status” custom field provides a sneak peek for it (explained ahead).
Where is it present
| Overall Requirement Coverage (indirectly) | Requirements List gadget | Traceability report |
|---|---|---|
|
|
|
How and issue/requirement can be covered
Understanding Sub-requirements
|
|
|
|---|
Requirements Hierarchy
Xray fully supports two level of requirements
- Traceability and Overall Coverage Reports/gadgets are limited to two levels
- It is possible to have more levels using one of the following combinations (although this hierarchy won’t be fully visible throughout Xray)
- Epic⬄Story relation + specific issue link
- Epic⬄Story relation + sub-task issues
|
|
|---|
Coverage - Which Tests are considered?
By default, the list of Tests that cover an issue/requirement is not dependent on the version, i.e. all “linked” Tests are considered.
If you wish to control which Tests are ”valid” for each version, then you must choose the “Use versioned Test Sets for Requirement Coverage” in the Requirement Coverage Xray settings.
- For each version, you must create a Test Set and assign it to the version you want, and only those Tests will be considered for the coverage calculation.
Coverage Status – Simplified overview
- OK All the tests associated with the issue (e.g. requirement) are PASSED
- NOK At least one test associated with the issue is FAILED
- NOT RUN At least one test associated with the issue is TODO, ABORTED or EXECUTING and there are no tests with status FAILED
- UNKNOWN At least one test associated with the issue is UNKNOWN and there are no Tests with status FAILED
- UNCOVERED The issue has no tests associated to it
Calculation of coverage status in given scope
- Obtain the list of Tests that cover the issue, directly or indirectly through sub-requirements
- Depends on the Coverage configuration in the Xray settings
- Calculate the Test status for all the Tests individually, in the given scope
- Considers Test Runs in version V (as a result of Test Executions in version V) or Test Runs in Test Plan TP (within Test Executions associated with Test Plan TP)
- If a specific Environment is also chosen, then only Test Runs from Test Executions with this Environment will be considered. In case no Environment is specified then all Test Executions are considered
- Calculate the coverage status mapped to the previously calculated Test statuses
- Compare the previously calculated coverage statuses together
Assessing the coverage status of an issue
Sub-requirements in Xray
The coverage status of the parent requirement will consider the sub-requirements status
You obtain the same value for the calculation of the status of the parent requirement if you consider that it is being covered by all the explicitly linked Tests and also the ones linked to sub-requirements.
Multiple examples of Coverage
Requirement has tests and its "UNCOVERED"
Test | Test status | Calculated coverage status |
- | - | No Tests cover the issue. |
Requirement has tests and its "Ok"
Test | Test status | Calculated coverage status |
1 | PASS | OK All Tests are passed (all contribute to the OK status of the requirement) |
2 | PASS | |
3 | PASS |
Requirement has tests and its "NOT RUN"
Test | Test status | Calculated coverage status |
1 | PASS | NOT RUN One of the Tests (3) is TODO which is mapped to NOT RUN, that has higher ranking than OK. |
2 | PASS | |
3 | TODO |
Requirement has tests and its NOK
Test | Test status | Calculated coverage status |
1 | PASS | NOK One of the Tests (3) is FAIL, which maps to NOK, that higher ranking than OK (of the mapped PASS tests) |
2 | PASS | |
3 | FAIL |
Sub-Requirement NOK
Test | Test status | Calculated coverage status |
1 | PASS | NOK One of the Tests (3b) is FAIL, thus subReq2 will be considered as NOK. Since it is NOK, then the parent requirement will be NOK. |
2a (subReq1) | PASS | |
3a (subReq2) | PASS | |
3b (subReq2) | FAIL | |
| | |
In this case, it is assumed that the parent requirement is NOK since one of the sub-requirements is NOK.
Sub-Requirement NOT RUN
Test | Test status | Calculated coverage status |
1 | PASS | NOT RUN One of the sub-requirements (subReq1) is NOT RUN, thus the calculated status, whenever doing the conjunction with the parent requirement status, will be NOT RUN. |
2a (subReq1) | TODO | |
3a (subReq2) | PASS | |
3b (subReq2) | PASS | |
| | |
Sub-Requirements is UNCOVERED
One Test directly linked to the parent requirement. Sub–requirement without tests.
Test | Test status | Calculated coverage status |
1 (parent requirement) | PASS | OK Since all sub-requirements are uncovered and the parent requirement is covered directly by one Test (1), which is currently PASS, then the calculated "OK" status will be based on that Test. Note: subReq1 is considered as being UNCOVERED |
- (subReq1) | - |
In this case, it is assumed that the parent requirement is completely validated by the Test directly linked to it.