This gadget provides a way to see the evolution of a group of Tests in some version of the system by taking into account some given Test Executions.
Aim
See the evolution/trend of the current status of a group of Tests, taking into account the results from those Tests in some given Test Executions.
This report allows the user to see the calculated status (e.g.. TODO, PASS, FAIL, etc) of the Tests per day, i.e. the amount of Tests by their current execution status, on a daily basis.
Some possible usage scenarios:
- see the progress of the calculated status of some Tests for some version;
- see the progress of the calculated status of some Tests for some version, on some specific Test Environment;
- see the progress of the calculated status of the Tests in a given Test Plan, taking into account the Test Executions related with the Test Plan.
How to use
Please note
The performance of this gadget will be mostly affect by the amount of Test Runs from the given Test Executions. Please use this gadget wisely.
Start by selecting the Tests and the Test Execution issues with:
- Saved Filter for Tests: A saved filter containing Test Issues
- Saved Filter for Test Executions: A saved filter containing Test Execution Issues
You can also restrict futher the Test Runs that will be used for the calculation of the values.
- Date Range: Custom or Time frame
- if "custom", the you need to provide the dates of the time period:
- Start date and End date: Only Test Runs with start date and end date between this range will be considered
- if "time frame", then it can be either:
- Current Week
- Last Week
- Current Nonth
- Last Month
- if "custom", the you need to provide the dates of the time period:
- Test Run Assignee: The Assignee of the Test Run
- Test Run Status: The status of the Test Run
- Test Environment: The Test Environment to take into account (i.e. only consider Test Runs from Test Executions made for this environment)
It is also possible to configure the:
- Custom Title: A custom title for the gadget
- Number of results: The number of results per page
- Refresh Interval: The rate at which the gadget will be updated
Please note
Please note that only the selected/filtered Test Runs from the given Test Executions are taken into account for the calculation of the status of the Tests for a given moment in time.
This means that only those Test Runs are considered. Therefore, as an example, if some Test was PASS (because you already had executed it somewhere in the past and the latest status of it was PASS) but you're only taking into account some more recent runs, then it will start by showing as TODO at the start of the chart.
How does it work
The Tests Evolution gadget alows you to see the evolution/trend of Tests over time, which may occur due to new executions made for those Tests or to changes made to the reported statuses in the existing Test Runs.
Threrefore, this gadget does its calculations based on the historical changes for the Test Runs being considered. Note that for a given Test Run you may or not change the Test Run statuses several times; this gadget will consider it.
It also takes into account the setting "Final statuses have precedence over non-final statuses", available under "Manage Test Statuses" in Xray settings.
General rules for the calculation of the status at a given day
- changes made over time to the status of a Test Run will be reflected in the status calculated for that Test (no matter if it's a final or non-final status)
- a status will "persist" (be valid) from the moment it has been recorded until the next status change in that Test Run or in another Test Run
- the "Started on" or the "Finished on" dates of the Test Runs are not taken into account for the calculation
- note that Test Runs are only created whenever you start executing them; therefore, if you create a Test Execution and don't execute the respective runs, then you won't see them being reflected in this gadget
"Final statuses have precedence over non-final statuses" checked
scenario | yesterday | today 10am | today 11am | |
---|---|---|---|---|
1 | TR1 | TR1 | - | Test Run |
PASS | TO DO | - | status change made in that day | |
PASS | TO DO | calculated Test status for that day | ||
2 | TR1 | TR1 | TR1 | Test Run |
PASS | FAIL | TO DO | status change made in that day | |
PASS | TO DO | calculated Test status for that day | ||
3 | TR1 | TR2 | - | Test Run |
PASS | TO DO | - | status change made in that day | |
PASS | PASS | calculated Test status for that day | ||
4 | TR1 | TR2 | Test Run | |
PASS | FAIL | status change made in that day | ||
PASS | FAIL | calculated Test status for that day | ||
5 | - | TR1 | Test Run | |
- | PASS | status change made in that day | ||
- | PASS | calculated Test status for that day |
"Final statuses have precedence over non-final statuses" unchecked
In this case, all statuses are considered independently if they are final or not. The key differentiator is that latest test runs are always considered, just by taking into account their creation date.
scenario | yesterday | today 10am | today 11am | |
---|---|---|---|---|
1 | TR1 | TR1 | TR1 | Test Run |
PASS | FAIL | TO DO | status change made in that day | |
PASS | TO DO | calculated Test status for that day | ||
2 | TR1 | TR2 | TR2 | Test Run |
PASS | FAIL | TO DO | status change made in that day | |
PASS | TO DO | calculated Test status for that day | ||
3 | - | TR1 | TR2 | Test Run |
- | FAIL | PASS | status change made in that day | |
- | PASS | calculated Test status for that day | ||
4 | TR1 | TR2 | - | Test Run |
TO DO | PASS | - | status change made in that day | |
TO DO | PASS | calculated Test status for that day |