Versions Compared

Old Version 1

changes.mady.by.user Unknown User (lfir)

Saved on

compared with

New Version Current

changes.mady.by.user Unknown User (lfir)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
UI Steps
UI Step

Subtle differences of statuses

  • Reported statuses for Test Run Steps and, implicitly, Test Runs are static; they are not dependent on any other variables or analysis criteria
  • The status of a Test or the coverage status of a “requirement” depends on how you analyze them(thus, implicitly on the Test Runs considered for that scope)

UI Step

Status of a Test Run

The state or result that was reported against a Test Run

  • Represents the status of the Test in the context of some Test Execution

Can be set explicitly/inline or automatically, based on the step level results (i.e. statuses)

You can generally visualize it in:


Execution screen itself (“execution status”)

Image Added


Test Runs panel (in Test issue screen, Test Execution issue screen)


Image Added

Traceability report

  • Test Runs List report/gadget

    • Image Removed


    Image Added


    UI Step

    Status of a Test

    The status of a Test is defined by how a given Test is “currently” performing in some scope/context

    Was it executed? Successfully? In which version?

    • Whenever speaking about the "status of a Test" we need to give it some additional context (e.g. "In which version?") since it depends on "where" and how you want to analyze it.
    • Depends on the latest consolidated results obtained for that Test, within some scope
      • Provides real-time info on the “status” of coverable issues (e.g. requirements, user stories, epics), for some context; thus, it has no relation whatsoever with the workflow status 
    • The same Test can be PASS for some context, FAIL for another one and even NOTRUN for another, for example


    Status of a Test – example with no environments

    The Test foo…

    StatusCondition
    ... is PASS on v3.0

    If the latest Test Run for that Test, in version 3.0, was PASS

    ... is PASS on v4.0

    If the latest Test Run for that Test, in version v4.0, was PASS

    ... is FAIL on v4.0

    If the latest Test Run for that Test, in version v4.0, was FAIL

    Info

    Test Run executed in version X means that the Test Run is part of a Test Execution assigned to version X


    Status of a Test – example with runs in many environments 

    The Test foo…

    StatusCondition
    ... is FAIL on v4.0 in environment X
    • If the latest Test Run for that Test, in version v4.0 in environment X, was FAIL
    ... is PASS on v3.0
    • If, for all  environments with Test Runs (for that Test) executed in v3.0, the latest Test Run was PASS
    ... is FAIL on v4.0
    • If, in any environment with Test Runs (for that Test) executed in v3.0, the latest Test Run was FAIL
    Info
    • Test Run executed in version X means that the Test Run is part of a Test Execution assigned to version X
    • Test Run executed in version X and environment X,  means that the Test Run is part of a Test Execution assigned to version X and Test Environment “X” 
    Info

    Statuses can be customized, please reach out your Jira administrator if you need to perform changes. 

    UI Step

    Status of a coverable issue (e.g. requirement)


    Info

    The project and the issue types that you want to be taken as requirements need to be configured by Configuring a Jira project to be used as a Requirements project and Issue Type Mapping.


    The status of a requirement is defined by how a given issue “currently is” from a quality standpoint, in some scope/context

    • The status of a requirement tells you information about its current state, from a quality perspective


    Is it covered with test cases? If so, has it been validated successfully? In which version?

    • Whenever speaking about the "status of a requirement" we need to give it some additional context (e.g. "In which version?") since it depends on "where" and how you want to analyze it
      • Depends on the latest consolidated results obtained for the Tests that cover  Test, within some context; thus, it’s not the workflow status 
    • The same “requirement” can be OK for some context, NOK for another one and even NOTRUN for another, for example


    Coverage Analysis – Brief examples

    "This requirement for the sensor is ..."

    StatusWhy
    ...UNCOVERED

    This requirement has no linked Tests

    ... is OK on v3.0

    All Tests related with that requirement are PASSING for v3.0

    ... is OK on v4.0

    All Tests related with that requirement are PASSING for v4.0

    ... is NOK on v3.0

    Some Test(s) related with that requirement are FAILING for v3.0

    ... is NOK on v4.0 in configuration XSome Test(s) related with that requirement are FAILING for v4.0 in configuration X
    Info

    Status can be customized, please reach out your Jira administrator if you need to perform changes. 

    UI Step

    Real status of requirements with in-context information


    Image RemovedImage Added


    Looking to the picture above we can view a the given spoken tests that we have spoken:

    • Test Status
    • The Requirement coverage under a specific context
    • The overall requirement coverage.

    ...