Reported statuses for Test Run Stepsand, implicitly, Test Runs are static; they are not dependent on any other variables or analysis criteria.
The status of a Test or the coverage status of a “requirement” depends on how you analyze them(thus, implicitly on the Test Runs considered for that scope).
UI Step
Status of a Test Run
The state or result that was reported against a Test Run
Represents the status of the Test in the context of some Test Execution
Can be set explicitly/inline or automatically, based on the step level results (i.e. statuses)
You can generally visualize it in:
Execution screen itself (“execution status”)
Test Runs panel (in Test issue screen, Test Execution issue screen)
Traceability report
Test Runs List report/gadget
Image Removed
Image Added
Image Added
Image Added
UI Step
Status of a Test
The status of a Test is defined by how a given Test is “currently” performing in some scope/context
Was it executed? Successfully? In which version?
Whenever speaking about the "status of a Test" we need to give it some additional context (e.g. "In which version?") since it depends on "where" and how you want to analyze it.
Depends on the latest consolidated results obtained for that Test, within some scope.
Provides real-time info on the “status” of coverable issues (e.g. requirements, user stories, epics), for some context; thus, it has no relation whatsoever with the workflow status.
The same Test can be PASS for some context, FAIL for another one and even NOTRUN for another, for example.
Status of a Test – example with no environments
The Test foo…
Status
Condition
... is PASS on v3.0
If the latest Test Run for that Test, in version 3.0, was PASS
... is PASSon v4.0
If the latest Test Run for that Test, in version v4.0, was PASS
... is FAIL on v4.0
If the latest Test Run for that Test, in version v4.0, was FAIL
Info
Test Run executed in version X means that the Test Run is part of a Test Execution assigned to version X
Status of a Test – example with runs in many environments
The Test foo…
Status
Condition
... is FAIL on v4.0 in environmentX
If the latest Test Run for that Test, in version v4.0 in environment X, was FAIL
... is PASS on v3.0
If, for all environments with Test Runs (for that Test) executed in v3.0, the latest Test Run was PASS
... is FAIL on v4.0
If, in any environment with Test Runs (for that Test) executed in v3.0, the latest Test Run was FAIL
Info
Test Run executed in version X means that the Test Run is part of a Test Execution assigned to version X
Test Run executed in version X and environment X, means that the Test Run is part of a Test Execution assigned to version X and Test Environment “X”
Info
Statuses can be customized, you can reach out your Jira administrator if you need to perform changes.
The status of a requirement is defined by how a given issue “currently is”, from a quality standpoint, in some scope/context
The status of a requirement tells you information about its current state, from a quality perspective
Is it covered with test cases? If so, has it been validated successfully? In which version?
Whenever speaking about the "status of a requirement" we need to give it some additional context (e.g. "In which version?") since it depends on "where" and how you want to analyze it
Depends on the latest consolidated results obtained for the Tests that cover Test, within some context; thus, it’s not the workflow status
The same “requirement” can be OK for some context, NOK for another one and even NOTRUN for another, for example
Coverage Analysis – Brief examples
This requirement for the scenario is ...
Status
Why
...UNCOVERED
This requirement has no linked Tests
... is OK on v3.0
All Tests related with that requirement are PASSING for v3.0
... is OKon v4.0
All Tests related with that requirement are PASSING for v4.0
... is NOK on v3.0
Some Test(s) related with that requirement are FAILING for v3.0
... is NOKon v4.0 in configuration X
Some Test(s) related with that requirement are FAILING for v4.0 in configuration X
Info
Status can be customized, you can reach out your Jira administrator if you need to perform changes.
UI Step
Real status of requirements with in-context information
Image RemovedImage Added
Looking to the picture above we can view a the given spoken tests that we have spoken: