Reported statuses for Test Run Stepsand, implicitly, Test Runs are static; they are not dependent on any other variables or analysis criteria
The status of a Test or the coverage status of a “requirement” depend on how you analyze them(thus, implicitly on the Test Runs considered for that scope)
Image Added
UI Step
Status of a Test
How a given Test is “currently” performing in some scope/context
Was it executed? Successfully? In which version?
Whenever speaking about the "status of a Test" we need to give it some additional context (e.g. "In which version?") since it depends on "where" and how you want to analyze it.
Depends on the latest consolidated results obtained for that Test, within some scope
Provides real-time info on the “status” of coverable issues (e.g. requirements, user stories, epics), for some context; thus, it has no relation whatsoever with the workflow status
The same Test can be PASS for some context, FAIL for another one and even NOTRUN for another, for example
Status of a Test – no environments
The Test foo…
Status
Condition
... is PASS on v3.0
If the latest Test Run for that Test, in version 3.0, was PASS
... is PASSon v4.0
If the latest Test Run for that Test, in version v4.0, was PASS
... is FAIL on v4.0
If the latest Test Run for that Test, in version v4.0, was FAIL
Info
Test Run executed in version X means that the Test Run is part of a Test Execution assigned to version X
Status of a Test – runs in many environments
The Test foo…
Condition
... is FAIL on v4.0 in environmentX
If the latest Test Run for that Test, in version v4.0 in environment X, was FAIL
... is PASS on v3.0
If, for all environments with Test Runs (for that Test) executed in v3.0, the latest Test Run was PASS
... is FAIL on v4.0
If, in any environment with Test Runs (for that Test) executed in v3.0, the latest Test Run was FAIL
Notes:
Test Run executed in version X means that the Test Run is part of a Test Execution assigned to version X
Test Run executed in version X and environment X, means that the Test Run is part of a Test Execution assigned to version X and Test Environment “X”
Quick Overview
Test – a test case template (not related with execution)
Test Run – represents a run of a given Test in the context of some scheduled Test Execution; it’s like an instance of the Test containing both the specification and the recorded results
“Requirement” – a coverable issue (i.e. an issue that you can cover with Tests); can be a any issue type, including Story, Epic and other, as long as they’re configurable to be handled as “requirements”. If you want, you can also configure Bug or similar issue types to be handled as “requirements” (i.e. coverable with test cases)