Page History
...
To calculate an issue coverage status for a specific system version, we “just” need to take into account the status of the related Tests for that same version. We’ll come back to this later on.
Calculation of the status for a given Test Run
The status of a given Test Run is an attribute that is often calculated automatically based on the respective recorded step statuses. You can also enforce a specific status for a Test Run, which in turn may implicitly enforce specific step statuses (e.g., setting a Test Run as "FAIL" can set all steps as "FAIL").
This calculation is made by comparing steps, following these rules:
- if a step status is mapped to a non-final, then Test Run status will be "EXECUTING"
- compare all the step statuses based on their order (steps at the bottom of the list will have higher ranking)
- the step status "PASS" has the lowest ranking
- the step status "FAIL" has the highest ranking
The order of the steps is irrelevant for the purpose of the overall Test Run status value.
Examples
The following table provides some examples given the Test Step Statuses configuration shown above.
- Obtain the test status mapped to each reported test step status; this is important as the actual test step statuses are not directly compared
- Compare all the the previously mapped test statuses together
- if any of these statuses (e.g., "PASS") is in turn mapped to the coverage status "OK", then the other status wins; if both are mapped to "OK" then the highest ranked wins
- if any of these statuses is "FAIL", then the Test Run status will be "FAIL"
- if any of these statuses is in turn mapped to the coverage status "NOK", then the Test Run status will be that one
- if any of theses statuses is final, then wins over non-final ones
- of these statuses, the status with the highest ranking wins
The order of the steps is irrelevant for the purpose of the overall Test Run status value.
Consequences:
- if any test step status is "FAIL" then the calculated status for the Test Run will be "FAIL"
- if any of the test steps "contributes negatively" (i.e., is mapped to a Test status associated with the NOK coverage), then the status of Test Run will correspond to the mapped Test status of that step
- the Test Run will have status "PASS" if all the steps are marked as "PASS"
- the calculated status for the Test Run will only be "EXECUTING" if there is at least one step in "EXECUTING" or "TODO" (or a similar custom test step status) and all other steps are in "PASS" or equivalent (i.e., associated to the "OK" coverage status)
Configuration Example 1
The following table provides some examples given the Test Step Statuses configuration shown above.
Example # | Statuses of the steps/contexts (the order of the steps/contexts is irrelevant) | Calculated value for the status of the Test Run | Why? |
|---|---|---|---|
| 1 |
| PASS | All steps are PASS, thus the joint value is PASS |
| 2 |
| EXECUTING | At least one step status (i.e. TODO) is mapped to a non-final Test status |
| 3 |
| FAIL | One of the step statuses (i.e. FAIL) has higher ranking than the other ones |
| 4 |
| FAIL | Since one of the steps is FAIL, then the run will be marked as FAIL. |
| 5 |
| FAIL | Since one of the steps is FAIL, then the run will be marked as FAIL. |
| 6 |
| FAIL | All mapped statuses map to a test status that in turn is associated to "NOK". Since one of them is FAIL, then the run will be marked as FAIL. |
Configuration Example 2
Let's consider the following configuration.
Example # | Statuses of the steps/contexts (the order of the steps/contexts is irrelevant) | Calculated value for the status of the Test Run | Why? |
|---|---|---|---|
| 1 |
| CUSTOM_PASS2 | We can see that both steps contribute in a "positive way" (i.e., they were successful as ultimately they are linked to successful coverage impact). Both statuses mapped to these test step statuses are associated with the "OK" coverage; as CUSTOM_PASS2 has higher ranking than CUSTOM_PASS, the run will be marked as "CUSTOM_PASS2". |
| 2 |
| CUSTOM_PASS2 | Similary to the previous example. Any status wins the "PASS" status. |
| 3 |
| CUSTOM_FAIL2 | We can see that both steps contribute in a "negative way" (i.e., they were not successful as ultimately they are linked to unsucessful coverage impact). Both statuses mapped to these test step statuses are associated with the "NOK" coverage; as CUSTOM_FAIL2 has higher ranking than CUSTOM_FAIL, the run will be marked as "CUSTOM_FAIL2" |
Example #
Statuses of the steps/contexts
(the order of the steps/contexts is irrelevant)
Calculated value for the status of the Test Run
Why?
- PASS
- PASS
- PASS
- PASS
- TODO
- PASS
- PASS
- FAIL
- PASS
- XPASS
- FAIL
- PASS
Since one of the steps is FAIL, then the run will be marked as FAIL.
- FAIL
- XPASS
- FAIL
. |
Calculation of the status for a given Test
It is possible to calculate the status of a Test either by Version or Test Plan, in a specific Test Environment or globally, taking into account the results obtained for all Test Environments.
...
- the flag "Final statuses have precedence over non-final statuses" shown in the calculation sections (enabled by default)
- the existence of Test Runs for different Test Environments, in case the analysis is made for "All Environments"
Calculate the status of some Test, in version V or Test Plan TP, for Test Environment TE
- This takes into account Test Runs in version V (as a result of Test Executions in version V) or Test Runs in Test Plan TP (within Test Executions associated with Test Plan TP)
- If Test Environment is chosen, then only Tests Runs on that Environment (e.g., TE) will be considered.
- If "Final statuses have precedence over non-final statuses" is true, then:
- final Test Run statuses will have higher ranking than non-final ones
- only the latest Test Run is taken into account based on its "finished on" date
- If "Final statuses have precedence over non-final statuses" is false, then:
- only the latest Test Run is taken into account based on its "created" date (i.e. the creation date of the related Test Run entity - this happens when a Test is added to the Test Execution)
Calculate the status of some Test, in version V or Test Plan TP, for "All Environments"
- calculate the Test status for each Test Environment, based on all the implicit Test Environments from the relevant Test Executions (i.e., Test Executions in version V or Test Executions associated with Test Plan TP)
- calculate the joint value for the Test status
- PASS has lowest ranking (i.e. for the calculated to be PASS, all calculated statuses must be PASS in the different Test Environments)
- if one is FAIL, then the calculated value will be FAIL
- otherwise, use the ranking of Test statuses
Examples
The following table provides some examples given the Test Statuses configuration shown above in the Managing Test Statuses section.
...
Example # | Statuses of the Test Runs (ordered by time of execution/creation, ascending) | Final statuses have precedence over non-final statuses | Calculated value for the status of the Test | Why? |
|---|---|---|---|---|
| 1a |
| true | PASS | Latest executed Test Run (2) having a final status was PASS. |
| 1b |
| false | TODO | Latest created Test Run (3) was TODO. |
| 2a |
| true | MYPASS2 | Latest executed final Test Runs on each environment were PASS, MYPASS2 and PASS respectively. Since MYPASS2 (3) has higher ranking then the calculated status will be MYPASS2. |
| 2b |
| false | TODO | Latest created Test Runs on each environment were PASS, TODO and PASS respectively. Since PASS has the lowest ranking, then TODO (3) will "win" and then the calculated status will be TODO |
| 3 |
| true | TODO | Latest created Test Runs on each environment were PASS, TODO and PASS respectively. Although Test Environment "env2" has only a non-final Test Run, since there is no other Run for that environment, then it will be considered as the calculated status for that environment. Since PASS has the lowest ranking, then TODO (3) will "win" and then the calculated status will be TODO. |
| 4 |
| true (or false) | FAIL | Latest executed (or created) final Test Runs on each environment were PASS, FAIL and PASS respectively. Since the calculated status for one of the environments is FAIL, then the calculated status will be FAIL. |
| 5 |
| true | MYPASS2 | Latest executed final Test Runs on each environment were PASS, MYPASS2 and MYFAIL respectively. MYPASS2 has higher ranking than the other ones, thus the overall calculated value will be MYPASS2. |
| 6 |
| false | MYFAIL | Latest created Test Runs on each environment were PASS, TODO and MYFAIL respectively. MYFAIL has higher ranking than the other ones, thus the overall calculated value will be MYFAIL. |
Calculation of the coverage status for a given issue
It is possible to calculate the test coverage status of a coverable issue either by Version or Test Plan, in a specific Test Environment or globally, taking into account the results obtained for all Test Environments.
...
Example # | Statuses of the related Tests (child issues, whenever present, appear as subReqX) | Calculated value for the coverage status of the issue | Why? |
|---|---|---|---|
| 1 |
| OK | All Tests are passed (it is similar to having just one virtual test that would be considered PASS and thus mapped to the OK status of the issue) |
| 2 |
| NOT RUN | One of the Tests (3) is TODO, which has higher ranking than PASS. |
| 3 |
| NOK | One of the Tests (3) is FAIL, which has higher ranking than PASS. |
| 4 |
| NOK | One of the Tests (3b) is FAIL, thus subReq2 will be considered as NOK. Since it is NOK, then the parent req which has higher ranking than PASS. |
| 5 |
| NOT RUN | One of the child issues (subReq1) is NOT RUN, thus the calculated status, whenever doing the conjunction with the parent issue status, will be NOT RUN. |
| 6 |
| OK | Since all child issues are uncovered and the parent issue is covered directly by one Test (1), which is currently PASS, then the calculated "OK" status will be based on that Test. |
Setup information for possible use cases
- I want to skip some Tests and proceed as they didn't exist
- Create a "Test Step Status" (e.g., "SKIP"), mapped to the Test Status "PASS"
- I want to fail a Test Run but I don't want to mark the requirement as being NOT OK because this failure can be discarded
- Create a "Test Status" (e.g., "FAIL_DISCARD") , non-final and mapped to the coverage status "UNKNOWN"; setting the status as non-final will give priority to other Test Runs you may have for that Test, If “Final Statuses have precedence over non-final” flag is enabled
- Create a "Test Step Status" (e.g., "IRRELEVANT_FAIL") and map it to the Test Status created in the previous step
- I want to always see, for a given Test, the status of Test based on the last run scheduled for it, no matter if it was completed (i.e. in a final status) or not
- Just uncheck the flag “Final Statuses have precedence over non-final”
- I want to execute some steps, set them as failed or passed, but I don't want them to reflect immediately in the status of the Test Run
- Create custom, non-final, Test statuses for passing and failure (e.g., "MYPASS", "MYFAIL"), mapped to the OK and NOK coverage statuses, respectively
- Create your own custom Test Step statuses for passed and failure (e.g., "PASS_CONTINUE" and FAIL_CONTINUE"), mapped to the previously created Test statuses
...