This gadget provides a way to see the evolution of a group of Tests in a particular version of the system by taking into account certain Test Executions.

Purpose

This gadget enables you to see, at a glance, the evolution/trend of the status of a group of Tests, taking into account the results of those Tests in certain Test Executions.

The calculated status (e.g., TODO, PASS, FAIL, etc.) of the Tests per day is shown.

Possible usage scenarios:

  • see the progress of the calculated status of certain Tests for a specific version
  • see the progress of the calculated status of certain Tests for a specific version on a specific Test Environment
  • see the progress of the calculated status of Tests in a given Test Plan, taking into account the Test Executions associated with the Test Plan.

How to use


Please note

The performance of this gadget will be mostly affect by the amount of Test Runs from the given Test Executions. Please use this gadget wisely.


Start by selecting the Tests and/or the Test Execution issues with:

  • Saved Filter for Tests: a saved filter containing Test Issues
  • Saved Filter for Test Executions: a saved filter containing Test Execution Issues

You can define one of these filters or both. If you just define the "Saved Filter for Tests", then the execution trend for those Tests will be shown, no matter in which Test Execution they were run. On the other hand, if you only define the "Saved Filter for Test Executions" then the execution trend will be shown for the Tests that are part of those Test Executions, in the scope of those Test Executions.

If both filters are defined, then the execution trend is shown just for the Tests contained within the "Saved Filter for Tests", taking into account the runs from the Test Executions identified by the "Saved Filter for Test Executions".


You can also restrict the Test Runs that will be used for the calculation of the values.

  • Date Range: either Custom, Time frame or Planned dates from Test Executions
    • if "Custom", then you need to provide the dates of the time period:
      • Start date and End date: only the Test Runs with start date and end date between this range will be considered
    • if "Time frame", then it can be either:
      • Current Week
      • Last Week
      • Current Month
      • Last Month
      • Last Days
    • if "Planned dates from Test Executions", then the minimum and maximum of begin and end dates, respectively, from the Test Executions are considered 
  • Test Run Assignee: the user assigned to the Test Run
  • Test Run Status: the status of the Test Run
  • Test Environment: the Test Environment to take into account (i.e., only consider the Test Runs from Test Executions made for this environment)

It is also possible to configure the:

  • Custom Title: a specific title for the gadget
  • Number of results: the number of results per page
  • Refresh Interval: how often the gadget is updated


Please note

Please note that only the selected/filtered Test Runs from the given Test Executions are taken into account in the calculation of the status of the Tests for any given moment in time.

This means that only those Test Runs are considered. If, for example, some Test was PASS (because you already had executed it somewhere in the past and the latest status of it was PASS) but you're only taking into account some more recent runs, then it will start by showing as TODO at the start of the chart.


  

How it works

Calculations are based on the historical changes for the Test Runs being considered. Note that for a given Test Run you may or not change the Test Run statuses several times; this gadget will consider it.

It also takes into account the Final statuses have precedence over non-final statuses setting, which is available under Manage Test Statuses in the Xray settings.


General rules for the calculation of the status for a given day

  • Changes made over time to the status of a Test Run will be reflected in the calculated status for that Test  (regardless of whether it's a final or a non-final status).
  • A status will persist (i.e., valid) from the moment it has been recorded until the next status change in that Test Run or in another Test Run.
  • The Started on or the Finished on dates of the Test Runs are not taken into account.
  • Test Runs are only created when you start executing them; therefore, if you create a Test Execution and don't execute the corresponding runs, then you won't see them being reflected in this gadget.

"Final statuses have precedence over non-final statuses" checked

ScenarioYesterdayToday 10amToday 11am
1

TR1

TR1

-Test Run
PASSTODO-status change made in that day
PASSTODOcalculated Test status for that day

2

TR1

TR1TR1Test Run
PASSFAILTODOstatus change made in that day
PASSTODOcalculated Test status for that day
3TR1TR2-Test Run
PASSTODO-status change made in that day
PASSPASScalculated Test status for that day
4

TR1TR2
Test Run
PASSFAIL
status change made in that day
PASSFAILcalculated Test status for that day
5-TR1
Test Run
-PASS
status change made in that day
-PASScalculated Test status for that day


"Final statuses have precedence over non-final statuses" unchecked

In this case, all statuses are considered independent of whether they are final or not. The key differentiator is that  latest test run is always considered, just by taking into account the creation date.

   

ScenarioYesterdayToday 10am

Today 11am


1

TR1

TR1

TR1Test Run
PASSFAILTODOstatus change made in that day
PASSTODOcalculated Test status for that day
2TR1TR2TR2Test Run
PASSFAILTODOstatus change made in that day
PASSTODOcalculated Test status for that day
3

-TR1TR2Test Run
-FAILPASSstatus change made in that day
-PASScalculated Test status for that day
4TR1TR2-Test Run
TODOPASS-status change made in that day
TODOPASScalculated Test status for that day




  • No labels