• Created by Unknown User (lfir), last modified on Sep 14, 2020

The different types of statuses

  • Reported statuses for Test Run Steps and, implicitly, Test Runs are static; they are not dependent on any other variables or analysis criteria.
  • The status of a Test or the coverage status of a “requirement” depends on how you analyze them (thus, implicitly on the Test Runs considered for that scope).

Status of a Test Run

The state or result that was reported against a Test Run

  • Represents the status of the Test in the context of some Test Execution

Can be set explicitly/inline or automatically, based on the step level results (i.e. statuses)

You can generally visualize it in:

  • Execution screen itself (“execution status”)
  • Test Runs panel (in Test issue screen, Test Execution issue screen)
  • Traceability report
  • Test Runs List report/gadget






Status of a Test

The status of a Test is defined by how a given Test is “currently” performing in some scope/context

Was it executed? Successfully? In which version?

  • Whenever speaking about the "status of a Test" we need to give it some additional context (e.g. "In which version?") since it depends on "where" and how you want to analyze it.
  • Depends on the latest consolidated results obtained for that Test, within some scope.
    • Provides real-time info on the “status” of coverable issues (e.g. requirements, user stories, epics), for some context; thus, it has no relation whatsoever with the workflow status.
  • The same Test can be PASS for some context, FAIL for another one and even NOTRUN for another, for example.


Status of a Test – example with no environments

The Test foo…

StatusCondition
... is PASS on v3.0

If the latest Test Run for that Test, in version 3.0, was PASS

... is PASS on v4.0

If the latest Test Run for that Test, in version v4.0, was PASS

... is FAIL on v4.0

If the latest Test Run for that Test, in version v4.0, was FAIL

Test Run executed in version X means that the Test Run is part of a Test Execution assigned to version X


Status of a Test – example with runs in many environments 

The Test foo…

StatusCondition
... is FAIL on v4.0 in environment X
  • If the latest Test Run for that Test, in version v4.0 in environment X, was FAIL
... is PASS on v3.0
  • If, for all  environments with Test Runs (for that Test) executed in v3.0, the latest Test Run was PASS
... is FAIL on v4.0
  • If, in any environment with Test Runs (for that Test) executed in v3.0, the latest Test Run was FAIL
  • Test Run executed in version X means that the Test Run is part of a Test Execution assigned to version X
  • Test Run executed in version X and environment X,  means that the Test Run is part of a Test Execution assigned to version X and Test Environment “X” 

Statuses can be customized, you can reach out your Jira administrator if you need to perform changes. 

Status of a coverable issue (e.g. requirement)


The project and the issue types that you want to be taken as requirements need to be configured by  Configuring a Jira project to be used as a Requirements project and Issue Type Mapping


The status of a requirement is defined by how a given issue “currently is”, from a quality standpoint, in some scope/context

  • The status of a requirement tells you information about its current state, from a quality perspective


Is it covered with test cases? If so, has it been validated successfully? In which version?

  • Whenever speaking about the "status of a requirement" we need to give it some additional context (e.g. "In which version?") since it depends on "where" and how you want to analyze it
    • Depends on the latest consolidated results obtained for the Tests that cover  Test, within some context; thus, it’s not the workflow status 
  • The same “requirement” can be OK for some context, NOK for another one and even NOTRUN for another, for example


Coverage Analysis – Brief examples

This requirement for the scenario is ...

StatusWhy
...UNCOVERED

This requirement has no linked Tests

... is OK on v3.0

All Tests related with that requirement are PASSING for v3.0

... is OK on v4.0

All Tests related with that requirement are PASSING for v4.0

... is NOK on v3.0

Some Test(s) related with that requirement are FAILING for v3.0

... is NOK on v4.0 in configuration XSome Test(s) related with that requirement are FAILING for v4.0 in configuration X

Status can be customized, you can reach out your Jira administrator if you need to perform changes. 

Real status of requirements with in-context information


Looking to the picture above we can view a the given spoken tests that we have spoken:

  • Test Status
  • The Requirement coverage under a specific context
  • The overall requirement coverage.
  • No labels