Versions Compared

Old Version 6

changes.mady.by.user Unknown User (lfir)

Saved on

compared with

New Version 7

changes.mady.by.user Unknown User (lfir)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
UI Steps
UI Step

Overview


The “latest”/consolidated status of the coverable issue (e.g. requirement) for a specific context​

​

  • Depends on where/how you’re analyzing your ”requirement"​
  • Generally, it represents the latest result obtained for that Test in that context​
  • More grained or high-level overview​
  • Can involve consolidating results from multiple Test Environments​
  • e.g. status of a Test for version 1.0, depends on consolidating the latest results for Chrome and Firefox




Info

the “Requirement Status” custom field provides a sneak peek for it (explained ahead)




UI Step

Where is it present 

Overall Requirement Coverage (indirectly)​Requirements List gadget​Traceability report​

UI Step

How and issue/requirement can be covered

UI Step

Understanding Sub-requirements

e



UI Step

Requirements Hierarchy


Xray fully supports two level of requirements​

  • Traceability and Overall Coverage Reports/gadgets are limited to two levels​

​

  • It is possible to have more levels using one of the following combinations (although this hierarchy won’t be fully visible throughout Xray)​
  • Epic⬄Story relation + specific issue link​
  • Epic⬄Story relation + sub-task issues​


Info

Coverage - Which Tests are considered?

By default, the list of Tests that cover an issue/requirement is not dependent on the version, i.e. all “linked” Tests are considered​

​

If you wish to control which Tests are ”valid” for each version, then you must choose the “Use versioned Test Sets for Requirement Coverage” in the Requirement Coverage Xray settings​

  • For each version, you must create a Test Set and assign it to the version you want, and only those Tests will be considered for the coverage calculation
UI Step

Coverage Status – Simplified overview​


  • OK  All the tests associated with the issue (e.g. requirement) are PASSED ​
  • NOK At least one test associated with the issue is FAILED​
  • NOT RUN  At least one test associated with the issue is TODO, ABORTED  or EXECUTING and there are no tests with status FAILED​
  • UNKNOWN  At least one test associated with the issue is UNKNOWN and there are no Tests with status FAILED​
  • UNCOVERED The issue has no tests associated to it​

​

UI Step

Calculation of coverage status in given scope​


  1. Obtain the list of Tests that cover the issue, directly or indirectly through sub-requirements
    1. Depends on the Coverage configuration in the Xray settings​
  2. Calculate the Test status for all the Tests individually, in the given scope
    1. Considers Test Runs in version V (as a result of Test Executions in version V) or Test Runs in Test Plan TP (within Test Executions associated with Test Plan TP)
    2. If a specific Environment is also chosen, then only Test Runs from Test Executions with this Environment will be considered. In case no Environment is specified then all Test Executions are considered​
  3. Calculate the coverage status mapped to the previously calculated Test statuses​
  4. Compare the previously calculated coverage statuses together​
UI Step

Assessing the coverage status of an issue​

UI Step

Sub-requirements in Xray​


The coverage status of the parent requirement will consider the sub-requirements status​

 



Info

you would obtain the same value for the calculation of the status of the parent requirement if you consider that it is being covered by all the explicitly linked Tests and also the ones linked to sub-requirements.​

UI Step

Multiple examples of Coverage

Requirement Uncovered



Info





Test​

Test status​

Calculated coverage status​

-​

-​

UNCOVERED​

​

No Tests cover the issue.​

Info

Requirement has tests and its "Ok"



Test​

Test status​

Calculated coverage status​

1​

PASS​

OK​

​

All Tests are passed​

(all contribute to the OK status of the requirement)​

2​

PASS​

3​

PASS​



Info

Requirement has tests and its  "NOT RUN"



Test​

Test status​

Calculated coverage status​

1​

PASS​

NOT RUN​

​

One of the Tests (3) is TODO which is mapped to NOT RUN, that has higher ranking than OK.​

2​

PASS​

3​

TODO​

Info

Requirement has tests and its   NOK



Test​

Test status​

Calculated coverage status​

1​

PASS​

NOK​

​

One of the Tests (3) is FAIL, which maps to NOK, that higher ranking than OK (of the mapped PASS tests)​

2​

PASS​

3​

FAIL​

Info

Sub-Requirement NOK​



Test​

Test status​

Calculated coverage status​

1​

PASS​

NOK​

​

One of the Tests (3b) is FAIL, thus subReq2 will be considered as NOK. Since it is NOK, then the parent requirement will be NOK.​

2a​

(subReq1)​

PASS​

3a​

(subReq2)​

PASS​

3b​

(subReq2)​

FAIL​

​

​

​

Info

In this case, it is assumed that the parent requirement is NOK  since one of the sub-requirements is NOK.​


Sub-Requirement NOT RUN​


Test​

Test status​

Calculated coverage status​

1​

PASS​

NOT RUN​

​

One of the sub-requirements (subReq1) is NOT RUN, thus the calculated status, whenever doing the conjunction with the parent requirement status, will be NOT RUN.​

2a​

(subReq1)​

TODO​

3a​

(subReq2)​

PASS​

3b​

(subReq2)​

PASS​

​

​

​


Sub-Requirements is UNCOVERED​

One Test directly linked to the parent requirement. Sub–requirement without tests.​


Test​

Test status​

Calculated coverage status​

1​

(parent requirement)​

PASS​

OK​

​

Since all sub-requirements are uncovered and the parent requirement is covered directly by one Test (1), which is currently PASS, then the calculated "OK" status will be based on that Test.​

​

Note: subReq1 is considered as being UNCOVERED​

​

-​

(subReq1)​

-​



Info

In this case, it is assumed that the parent requirement is completely validated by the Test directly linked to it.​