The different types of statuses
- Reported statuses for Test Run Steps and, implicitly, Test Runs are static; they are not dependent on any other variables or analysis criteria.
- The status of a Test or the coverage status of a “requirement” depends on how you analyze them (thus, implicitly on the Test Runs considered for that scope).
Status of a Test Run
The state or result that was reported against a Test Run
- Represents the status of the Test in the context of some Test Execution
Can be set explicitly/inline or automatically, based on the step level results (i.e. statuses)
You can generally visualize it in:
- Execution screen itself (“execution status”)
- Test Runs panel (in Test issue screen, Test Execution issue screen)
- Traceability report
- Test Runs List report/gadget
Status of a Test
The status of a Test is defined by how a given Test is “currently” performing in some scope/context
Was it executed? Successfully? In which version?
- Whenever speaking about the "status of a Test" we need to give it some additional context (e.g. "In which version?") since it depends on "where" and how you want to analyze it.
- Depends on the latest consolidated results obtained for that Test, within some scope.
- Provides real-time info on the “status” of coverable issues (e.g. requirements, user stories, epics), for some context; thus, it has no relation whatsoever with the workflow status.
- The same Test can be PASS for some context, FAIL for another one and even NOTRUN for another, for example.
Status of a Test – example with no environments
The Test foo…
Status | Condition |
... is PASS on v3.0 | If the latest Test Run for that Test, in version 3.0, was PASS |
... is PASS on v4.0 | If the latest Test Run for that Test, in version v4.0, was PASS |
... is FAIL on v4.0 | If the latest Test Run for that Test, in version v4.0, was FAIL |
Test Run executed in version X means that the Test Run is part of a Test Execution assigned to version X
Status of a Test – example with runs in many environments
The Test foo…
Status | Condition |
---|---|
... is FAIL on v4.0 in environment X |
|
... is PASS on v3.0 |
|
... is FAIL on v4.0 |
|
- Test Run executed in version X means that the Test Run is part of a Test Execution assigned to version X
- Test Run executed in version X and environment X, means that the Test Run is part of a Test Execution assigned to version X and Test Environment “X”
Statuses can be customized, you can reach out your Jira administrator if you need to perform changes.
Status of a coverable issue (e.g. requirement)
The project and the issue types that you want to be taken as requirements need to be configured by your JIRA Admin.
The status of a requirement is defined by how a given issue “currently is”, from a quality standpoint, in some scope/context
- The status of a requirement tells you information about its current state, from a quality perspective
Is it covered with test cases? If so, has it been validated successfully? In which version?
- Whenever speaking about the "status of a requirement" we need to give it some additional context (e.g. "In which version?") since it depends on "where" and how you want to analyze it
- Depends on the latest consolidated results obtained for the Tests that cover Test, within some context; thus, it’s not the workflow status
- The same “requirement” can be OK for some context, NOK for another one and even NOTRUN for another, for example
Coverage Analysis – brief examples
This requirement for the scenario is ...
Status | Why |
---|---|
...UNCOVERED | This requirement has no linked Tests |
... is OK on v3.0 | All Tests related with that requirement are PASSING for v3.0 |
... is OK on v4.0 | All Tests related with that requirement are PASSING for v4.0 |
... is NOK on v3.0 | Some Test(s) related with that requirement are FAILING for v3.0 |
... is NOK on v4.0 in configuration X | Some Test(s) related with that requirement are FAILING for v4.0 in configuration X |
Status can be customized, you can reach out your Jira administrator if you need to perform changes.
Real status of requirements with in-context information
Looking to the picture above we can view a the given spoken tests that we have spoken:
- Test Status
- The Requirement coverage under a specific context
- The overall requirement coverage.