Purpose

This report enables you to extract a list of Test Runs related to the input issues, so you can use it for analysis of trends, current testing status, or process this information to generate some metrics, for example, or even share it with someone else that hasn't access to Jira.

Possible usage scenarios:

• see all the Test Runs for a given Test Execution, Test Plan, Test, Test Set, or Story
• see the failed Test Runs and understand what happened and its impacts at high-level
• see the defects linked to failed Test Runs
• see the requirements linked to failed Test Runs
• see the failed Test Runs that have no defects
• track the Test Runs that are taking most time

Output Example(s)

The following table shows an example of the columns/rows you should expect, in this for the "Test Runs" sheet.

In more depth...

How to use

This report can be generated from different places/contexts, including:

• Issue view screen
• Issue search page (main search page or as bulk operation)

Source data

This report is applicable to:

• 1 or more Test Plan issues
• 1 or more Test Set issues
• 1 or more Test Execution issues
• 1 or more Test issues
• 1 or more Story issues
• a combination of the previous

Please note: to avoid redundancy in the output, we recommend not mixing source types, especially if the entities are also linked with each other (e.g. passing the issue keys for Stories and Test Executions linked to them to the template at the same time)

Output format

The standard output format is .XLSX, so you can open it in Microsoft Excel, Google Sheets, and other tools compatible with this format. From those tools you may be able to generate a .CSV file.

Report assumptions

The template has a set of assumptions that you make sure that your Jira/Xray environment complies with:

1. Issue types having these names
   1. "Test", "Test Set", "Test Plan", "Test Execution", "Story"

If any of these assumptions is not met, you need to update the template accordingly.

Usage examples

Export Test Runs associated with a given release

1. from the Issue Navigator/Search, search by the release (i.e., "fixVersion") of your project (e.g., "BOOK") and then use bulk export

   example of JQL expression to use

   project = BOOK and fixVersion = 1.0 and issuetype = "Test Execution"

Export Test Runs obtained in the context of a given Test Plan

1. open the Test Plan issue and export it using this template

Export Test Runs obtained on a given Test Environment

1. from the Issue Navigator/Search, search by Test Executions assigned to that Test Environment  (e.g., "chrome") and then use bulk export

   example of JQL expression to use

   project = BOOK and issuetype = "Test Execution" and  "Test Environments" = chrome

Understanding the report

The report shows information about the Test Runs for the input issues provided.

Layout

The report is composed by several sheets. The information on the "Test Runs" and "Test Runs with iterations" is mostly the same; please see details ahead.

By default, and to avoid overload/redundancy of information, only the "Test Runs" and the "Input Issues" sheets will be rendered; you can change this behavior on the template (more info ahead).

"Test Runs" sheet

This sheet will present a line per each Test Run, no matter the type of Test Run.

"Test Runs including iterations" sheet

This sheet will present:

• a line per each Test Run that is non data-driven (i.e., a standard run)
• for data-driven tests, a line per each iteration (i.e., data row) of the Test Run

Therefore, this sheet is like a superset of the "Test Runs" sheet, where data-driven test runs are expanded to multiple rows.

The fields displayed in this sheet are similar to the previous one.

"Input Issues" sheet

This sheet list the issues that were used as input for generating the report.

Customizing the report

Sections that can be hidden or shown

The report has some variables/flags that can be used to show or hide some sections whose logic is already implemented in the template.

These variables are defined at the top of each sheet, at the report template; the variables are scoped just to the current sheet.

On the template, use one of these values for flag type of variables:

• 0: to hide a section
• 1: to show a section

The format for other types of variables is detailed ahead.

Adding or removing information to/from the report

As this report is column based, if some columns are not relevant to you, you should be able to simply delete them. Make sure that no temporary variables are created in the cells of this column that are used in other subsequent columns.

To add additional information, usually we're thinking of adding:

• fields of the Test Run itself
• fields of the Test associated with the Test Run
• fields of the Test Execution associated with the Test Run

Eventually, also:

• fields of the Test Plan(s) associated with the Test Execution that is associated with the Test Run
• fields of the covered issue(s) associated with the Test that is associated with the Test Run

The later may be harder to implement, so we won't consider them here.

Exercise: add a field from the related Test issue 

Let's say we have a "Severity" field on the Test issue that is connected to each Test Run, and that we want to show it on the report.

We can copy the column "Test Summary" and adapt it.

1. insert column
2. on the "Test Runs" sheet,
   1. copy "Test Summary" (i.e., insert a column  next to it and copy the values from the existing "Test Summary" column)
   2. change
      1. ${TestRuns[n].Summary} to ${TestRuns[n].Severity}
      2. ${TestExecutions[j].TestRuns[n].Severity} to ${TestExecutions[j].TestRuns[n].Severity}
3. follow a similar approach for the "Test Runs including iterations" sheet

Exercise: add total Evidence count

Let's say we want to know the total count of evidence items for each run and iteration. You can add the column next to the "Comment" one, then the code logic is similar to the defect count. Please see the snippet below (this exact syntax is for the "Test Runs" tab):

${set(totalStepEvidenceCount, 0)}
${set(totalEvidenceCount, 0)}
#{if (%{!${TestExecutions[j].TestRuns[n].IsDataDriven}})}
#{for m=TestExecutions[j].TestRuns[n].TestStepsCount}
#{for l=TestExecutions[j].TestRuns[n].TestSteps[m].EvidencesCount}
${set(totalStepEvidenceCount,%{${totalStepEvidenceCount} + 1 })}
#{end}
#{end}
${set(totalEvidenceCount,%{${totalStepEvidenceCount} + ${TestExecutions[j].TestRuns[n].ExecutionEvidencesCount} })}
#{end}
#{if (%{${TestExecutions[j].TestRuns[n].IsDataDriven}})}
#{for m=TestExecutions[j].TestRuns[n].IterationsCount}
#{for k=TestExecutions[j].TestRuns[n].Iterations[m].TestStepsCount}
#{for l=TestExecutions[j].TestRuns[n].Iterations[m].TestSteps[k].EvidencesCount}
${set(totalStepEvidenceCount,%{${totalStepEvidenceCount} + 1 })}
#{end}
#{end}
${set(totalEvidenceCount,%{${totalStepEvidenceCount} + ${TestExecutions[j].TestRuns[n].ExecutionEvidencesCount} })}
#{end}
#{end}
${totalEvidenceCount}

Keep in mind that the core part of the path to the target (i.e. "TestExecutions[j].TestRuns[n]." above) is dependent on the source issue type, so you will need to adjust it based on the row in the template. You can refer to other cells in the same row for the path syntax.

For the "Test Runs including Iterations" tab, you will only need the content of one of the two "if" loops for each row and, for the data-driven rows, you will need to drop the additional "for" loop (i.e. "#{for m=TestExecutions[j].TestRuns[n].IterationsCount}" from the second "if" loop in the snippet above).

Performance

Performance can be impacted by the information that is rendered and by how that information is collected/processed.

The number of Test Runs, depending on scenarios, can be considerably high, especially with CI/CD. As this report sums up quite information, please use it wisely.

Data-driven tests may also add an overhead, as iterations need to be individually processed, for collecting all the reported/linked defects for example.

Known limitations

• Test Runs comments are not formatted
• Gherkin Scenario Outlines are not considered as data-driven (i.e., only one Test Run will appear)