Versions Compared

Old Version 5

changes.mady.by.user Sergio Freire

Saved on

compared with

New Version 6

changes.mady.by.user Sergio Freire

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Let’s start by detailing the different possible values Test, Test Step and coverage statuses. In the end, we’ll see how they’ll impact on the calculation of the coverage status of a issue in some specific version or Test Plan.

Overview of statuses

Whenever talking about statuses, we may be talking about statuses of requirements, Tests, Test Runs and Test Steps.

...

Info
titlePlease note

Whenever we're speaking about the status associated with a requirement or with a Test (and even with a Test Set), we may be talking about different things:

  • (coverage) status for some version, taking into account executions made for that version of the Tests that validate the requirement
  • workflow status associated to the requirement issue (e.g., "New", "In Progress", "Closed")


The current page details the first one, i.e. the status of the entities based on the executions made in some context.

Test status

The status of a Test tells you information about its current consolidated state (e.g. latest record result, if existent). Was it is executed? Successfully? In which version?

...

The status (i.e. result) of a Test Run is an attribute of the Test Run (a “Test Run” is an instance of a Test and is not a Jira issue) and is the one taken into account to assess the coverage status of the coverable issue.


Managing Test Statuses

Creating new Test (Run) statuses may be done in the Global Settings: Test Statuses configuration section of Xray.

...

This may be used, when users prefer to take into account only the last final/complete recorded result and want to discard Test Runs that are in an intermediate status (e.g. EXECUTING, TODO).

Test Step statuses

The status of a Test Step indicates the result obtained for that step for some Test Run. 

...

Test Step statusTest status
PASSPASS
TODOTODO
EXECUTINGEXECUTING
FAILFAIL
custom

custom


Managing Test Step Statuses

Creating new Test Step statuses may be done in the Global Settings: Test Step Statuses configuration section of Xray.

...

Note that native Test Step statuses can’t be modified nor deleted.

Test Coverage statuses

The (coverage) status of a coverable issue tells you information about its current state, from a quality perspective. Is it covered with test cases? If so, has it been validated successfully? In which version?

...

You can see that in order to calculate an issue coverage status, for some specific system version, we “just” need to take into account the status of the related Tests for that same version. We’ll come back to this later on.

Calculation of the status for a given Test Run

The status of a given Test Run is an attribute that most of the times is calculated automatically on the respective recorded step statuses. User can also enforce a specific status for a Test Run, which in turn may implicitly enforce specific step statuses (e.g. setting a Test Run as "FAIL" can set all steps as "FAIL"). 

...

The order of the steps is indifferent for the purpose of the overall Test Run status value.



Examples

The following table provides some examples given the Test Step Statuses configuration shown above.

...

Example #

Statuses of the steps/contexts

(the order of the steps/contexts is irrelevant)

Calculated value for the status of the Test RunWhy?
1
  • PASS
  • PASS
  • PASS
PASSAll steps are PASS, thus the joint value is PASS
2
  • PASS
  • TODO
  • PASS
EXECUTINGAt least one step status (i.e. TODO) is mapped to a non-final Test status
3
  • PASS
  • FAIL
  • PASS
FAILOne of the step statuses (i.e. FAIL) has higher ranking than the other ones
4
  • XPASS
  • FAIL
  • PASS
FAIL

XPASS has higher ranking than the other ones, thus the overall calculated value is based on the mapping for that Test Step status.


5
  • FAIL
  • XPASS
  • FAIL
FAILXPASS has higher ranking than the other ones, thus the overall calculated value is based on the mapping for that Test Step status.


Calculation of the status for a given Test

It is possible to calculate the status of a Test either by Version or Test Plan, in a specific Test Environment or globally, taking into account the results obtained for all Test Environments.

...

  • the flag "Final statuses have precedence over non-final statuses" shown in the calculation sections (enabled by default)
  • the existence of Test Runs for different Test Environments, in case the analysis is made for "All Environments"

Calculate the status of some Test, in version V or Test Plan TP, for Test Environment TE

  1. This takes into account Test Runs in version V (as a result of Test Executions in version V) or Test Runs in Test Plan TP (within Test Executions associated with Test Plan TP)
  2. If Test Environment is chosen, then only Tests Runs on that Environment (e.g. TE) will be considered
  3. If "Final statuses have precedence over non-final statuses" is true, then:
    • final Test Run statuses will have higher ranking than non-final ones
    • only the latest Test Run is taken into account based on its "finished on" date  
  4. If "Final statuses have precedence over non-final statuses" is false, then:
    • only the latest Test Run is taken into account based on its "created" date (i.e. the creation date of the related Test Execution)

Calculate the status of some Test, in version V or Test Plan TP, for "All Environments"

  1. calculate the Test status for each Test Environment, based on all the implicit Test Environments from the relevant Test Executions (i.e. Test Executions in version V or Test Executions associated with Test Plan TP)
  2. calculate the joint value for the Test status
    1. PASS has lowest ranking (i.e. for the calculated to be PASS, all calculated statuses must be PASS in the different Test Environments)
    2. if one is FAIL, then the calculated value will be FAIL
    3. otherwise, use the ranking of Test statuses


Examples

The following table provides some examples given the Test Statuses configuration shown above in the Understanding coverage and the calculation of Test and requirement statuses Managing Test Statuses section.


Example #

Statuses of the Test Runs

(ordered by time of execution/creation, ascending)

Final statuses have precedence over non-final statusesCalculated value for the status of the TestWhy?
1a
  1. PASS
  2. PASS
  3. TODO
truePASSLatest executed Test Run (2) having a final status was PASS.
1b
  1. PASS
  2. PASS
  3. TODO
falseTODOLatest created Test Run (3) was TODO.
2a
  1. PASS (env1)
  2. MYPASS2 (env2)
  3. TODO (env2)
  4. PASS (env3)
trueXPASS

Latest executed final Test Runs on each environment were PASS, MYPASS2 and PASS respectively.

Since MYPASS2 (3) has higher ranking then the calculated status will be MYPASS2.


2b
  1. PASS (env1)
  2. MYPASS2 (env2)
  3. TODO (env2)
  4. PASS (env3)
falseXPASS

Latest created Test Runs on each environment were PASS, TODO and PASS respectively.

Since PASS has the lowest ranking, then TODO (3) will "win" and then the calculated status will be TODO

3
  1. PASS (env1)
  2. TODO (env2)
  3. PASS (env3)
trueTODO

Latest created Test Runs on each environment were PASS, TODO and PASS respectively.

Although Test Environment "env2" has only a non-final Test Run, since there is no other Run for that environment, then it will be considered as the calculated status for that environment.

Since PASS has the lowest ranking, then TODO (3) will "win" and then the calculated status will be TODO.

4
  1. PASS (env1)
  2. FAIL (env2)
  3. PASS (env3)
true (or false)FAIL

Latest executed (or created) final Test Runs on each environment were PASS, FAIL and PASS respectively.

Since the calculated status for one of the environments is FAIL, then the calculated status will be FAIL.

5
  1. PASS (env1)
  2. MYPASS2 (env2)
  3. TODO (env2)
  4. MYFAIL (env3)
trueMYPASS2

Latest executed final Test Runs on each environment were PASS, MYPASS2 and MYFAIL respectively.

MYPASS2 has higher ranking than the other ones, thus the overall calculated value will be MYPASS2.


6
  1. PASS (env1)
  2. MYPASS2 (env2)
  3. TODO (env2)
  4. MYFAIL (env3)
falseMYFAIL

Latest created Test Runs on each environment were PASS, TODO and MYFAIL respectively.

MYFAIL has higher ranking than the other ones, thus the overall calculated value will be MYFAIL.


Calculation of the coverage status for a given issue

It is possible to calculate the test coverage status of a coverable issue either by Version or Test Plan, in a specific Test Environment or globally, taking into account the results obtained for all Test Environments.

...

  • indirectly, the flag "Final statuses have precedence over non-final statuses" (enabled by default)
  • the existence of Test Runs for different Test Environments, in case the analysis is made for "All Environments"

Test Coverage Hierarchy

Sometimes you may have parent requirements and sub-requirements, for example. In general, you may have parent issues and "child" issues, both of them being handled as coverable issues.

...

Info
titleRationale

Even if you are using the Test Coverage Hierarchy related features, when you have tests that are directly linked to the parent issue, Xray assumes that you are validating the parent issue directly. Thus, it's irrelevant if the child issues are uncovered by tests or not.


Examples

The following table provides some examples given the Test Statuses configuration shown above in the Understanding coverage and the calculation of Test and requirement statuses Managing Test Statuses section.

Example #

Statuses of the related Tests

(child issues, whenever present, appear as subReqX)

Calculated value for the coverage status of the issueWhy?
1
  1. PASS
  2. PASS
  3. PASS
OKAll Tests are passed (it is similar to having just one virtual test that would be considered PASS and thus mapped to the OK status of the issue)
2
  1. PASS
  2. PASS
  3. TODO
NOT RUNOne of the Tests (3) is TODO, which has higher ranking than PASS.
3
  1. PASS
  2. PASS
  3. FAIL

NOK

One of the Tests (3) is FAIL, which has higher ranking than PASS.
4
  1. PASS
  2. subReq1 => OK
    1. PASS
  3. subReq2 => NOK
    1. PASS
    2. FAIL

NOK

One of the Tests (3b) is FAIL, thus subReq2 will be considered as NOK. Since it is NOK, then the parent req which has higher ranking than PASS.
5
  1. PASS
  2. subReq1 => NOTRUN
    1. TODO
  3. subReq2 => OK
    1. PASS
    2. PASS
NOT RUNOne of the child issues (subReq1) is NOT RUN, thus the calculated status, whenever doing the conjunction with the parent issue status, will be NOT RUN.
6
  1. PASS
  2. subReq1 => UNCOVERED
    1. (no Tests associated)
  3. subReq2 => UNCOVERED
    1. (no Tests associated)
OKSince all child issues are uncovered and the parent issue is covered directly by one Test (1), which is currently PASS, then the calculated "OK" status will be based on that Test.

Setup information for some possible use cases

  1. I want to skip some Tests and proceed as they didn't exist
    1. create a "Test Step Status"  (e.g. "SKIP"), mapped to the Test Status "PASS"
  2. I want to fail a Test Run but I don't want to mark the requirement as being NOT OK because this failure can be discarded
    1. create a "Test Status" (e.g. "FAIL_DISCARD") , non-final and mapped to the coverage status "UNKNOWN"; setting the status as non-final will give priority to other Test Runs you may have for that Test, If “Final Statuses have precedence over non-final” flag is enabled
    2. create a "Test Step Status" (e.g. "IRRELEVANT_FAIL") and map it to the Test Status created in the previous step
  3. I want to always see, for a given Test, the status of Test based on the last run scheduled for it, no matter if it was completed (i.e. in a final status) or not
    1. just uncheck the flag “Final Statuses have precedence over non-final”
  4. I want to execute some steps, set them as failed or passed, but I don't want them to reflect immediately in the status of the Test Run
    1. create custom, non-final, Test statuses for passing and failure (e.g. "MYPASS", "MYFAIL"), mapped to the OK and NOK coverage statuses, respectively
    2. create your own custom Test Step statuses for passed and failure (e.g. "PASS_CONTINUE" and FAIL_CONTINUE"), mapped to the previously created Test statuses 

...