--- title: "Document Generator Template: Test Runs List Report" canonical: "https://docs.getxray.app/space/XRAYCLOUD/74354670/Document%20Generator%20Template%3A%20Test%20Runs%20List%20Report" format: markdown --- > Macro (ui-expand) > > > Macro (toc) # Introduction The Test Runs List Report enables you to extract a list of [Test Runs](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565109) related to the input Issues, so you can use it for analysis of trends, and current testing status, or process this information to generate some metrics, for example, or even share it with someone else that has no access to Jira. Possible usage scenarios: - See all the Test Runs for a [Test Execution](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565117), [Test Plan](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44578481), Test, or Story. - See the failed Test Runs and understand what happened and its impacts at a high level. - See the Defects linked to failed Test Runs. - See the Requirements linked to failed Test Runs. - See the failed Test Runs that have no Defects. - Track the Test Runs that are taking the most time. ## Output Example(s) The following table (Figure 1) shows an example of the columns/rows you should expect for the *Test Runs* *List* sheet. ![Figure 1 - Report](media://66353d47-b929-4279-a59b-48406666bb31) You can see a more detailed table below. | **Test Key** | **Test Summary** | **Test Type** | **Test Label(s)** | **Test Component(s)** | **Test Priority** | **Total Parameters** | **Parameters** | **Requirements** | **Test Plan Key** | **Test Execution Key** | **FixVersion(s) ** | **Revision** | **Execution Status** | **TE assignee** | **TE planned begin date** | **TE planned end date** | **Assignee** | **Executed by** | **Test Environment(s)** | **Started at** | **Finished at** | **Elapsed** | **Elapsed (sec)** | **Comment** | **Total Defects** | **Defects** | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | XT-506 | Accessibility Validations | Generic | Label1,Label3 | Visits | Medium | 0 | | XT-507 | EWB-8 | EWB-315 | 1.0 | 123 | PASSED | Cristiano Cunha | | | Cristiano Cunha | Sergio Freire | Production, chrome | 11/11/2022 04:18:47 | 11/11/2022 04:18:47 | 00:00:00 | 0,00 | all good | 0 | | | XT-552 | Accessibility - getViolations | Generic | | | Medium | 0 | | XT-507 | EWB-8 | EWB-315 | 1.0 | 123 | EXECUTING | Cristiano Cunha | | | Cristiano Cunha | Sergio Freire | Production, chrome | 19/09/2022 06:00:00 | | 00:00:00 | 0,00 | | 0 | | | EWB-321 | Data-driven manual Test | Manual | | Login, 5Modules | Medium | 6 | Key: username  Value: user1 _________  Key: password  Value: pass1 _________  Key: username  Value: user2 _________  Key: password  Value: pass2 _________  Key: username  Value: user3 _________  Key: password  Value: pass3 _________ | EWB-325,EWB-326 | EWB-8 | EWB-315 | 1.0 | 123 | FAILED | Cristiano Cunha | | | Cristiano Cunha | Sergio Freire | Production, chrome | 08/11/2022 04:38:10 | 08/11/2022 04:38:29 | 00:00:19 | 19,00 | several issues found | 3 | EWB-324,EWB-323,EWB-322 | # Operations ## Using the Report ### Access #### From the Issue Navigator/Search Screen > Macro (ui-steps) > > > Macro (legacy-content) > > > Macro (legacy-content) > > > > Macro (legacy-content) ### Source Data This report is applicable to: - One or more Test Plan Issues - 1 or more Test Execution Issues. - One or more Test Issues. - One or more Story Issues. - A combination of the previous. > ⚠️ To avoid redundancy in the output, we recommend not mixing source types - especially if the entities are also linked with each other (e.g. passing the Issue keys for Stories and Test Executions linked to them to the Template at the same time). ### Output Format The standard output format is XLSX, so you can open it in Microsoft Excel, Google Sheets, and other tools compatible with this format.  From those tools, you may be able to generate a CSV file. ### Report Assumptions The report/Template has a set of assumptions that you have to make sure your Jira/Xray environment complies with: - Issue Types having these names: "Test", "Test Plan", "Test Execution", "Story". - Coverage between a Test and the covered item (e.g., Story Issue) using the Issue link type "Tests". If any of these assumptions are not met, you need to update the Template accordingly. ### Usage Examples #### Exporting the Test Runs Associated with a Release From the Issue Navigator/Search screen, search by the release (i.e., "fixVersion") of your project (e.g., "BOOK"), and then use bulk export. ##### **Example of JQL expression to use** ``` project = BOOK and fixVersion = 1.0 and issuetype = "Test Execution" ``` #### Exporting Test Runs Obtained in the Context of a Test Plan Open the Test Plan Issue and export it using the Test Runs List report. #### Exporting Test Runs Obtained on a Test Environment From the Issue Navigator/Search screen, search by Test Executions assigned to that [Test Environment](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565129)  (e.g., "chrome"), and then use bulk export. ##### **Example of a JQL expression to use** ``` project = BOOK and issuetype = "Test Execution" and testEnvironments = chrome ``` ## Understanding the Report The Test Runs List report shows information about the Test Runs for the input Issues provided. ### Layout The report is composed of several sheets. The information on the *Test Runs* and *Test Runs with Iterations* is mostly the same; please see the details below. By default, and to avoid overload/redundancy of information, only the *Test Runs* and the *Input Issues* sheets will be rendered; you can change this behavior on the Template (more info below). #### Test Runs Sheet This sheet will present a line for each Test Run, no matter the Type of Test Run. | **Column** | **Description** | | --- | --- | | **Test Key** | Issue key of the Test associated with this Test Run | | **Test Summary** | Summary of the Test associated with this Test Run | | **Test Type** | Test Type of the Test associated with this Test Run (e.g., Manual, Cucumber, Generic) | | **Test Label(s)** | Labels of the Test associated with the Test Run, delimited by a comma (e.g., "UI, selenium") | | **Test Component(s)** | Component field of the Test associated with the Test Run, delimited by a comma (e.g., "core, backend") | | **Test Priority** | Priority field of the Test associated with the Test Run, delimited by a comma (e.g., "Major") | | **Total Parameters** | Number of Test [Parameters](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44565547), for [Parameterized Tests](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44566351) | | **Parameters** | List of Test Parameters, for Parameterized Tests (e.g., key:   value: ") | | **Requirements** | List of covered items (e.g., requirements, stories) by the Test associated with the Test Run, delimited by a comma (e.g., "CALC-1") | | **Test Plan Key** | Issue key of the Test Plan(s), associated with the Test Execution, that in turn is associated with the Test Run | | **Test Execution Key** | Issue key of the Test Execution associated with the Test Run | | **FixVersion(s)** | FixVersion field of the Test Execution associated with the Test Run | | **Revision** | Revision field of the Test Execution associated with the Test Run | | **Execution Status** | The status/result reported for this Test Run | | **TE assignee** | The display name of the assignee of the Test Execution associated with the Test Run | | **TE planned the beginning date** | Planned date for starting the Test Execution, in the format "dd/MM/yyyy" | | **TE planned end date** | Planned date for finishing the Test Execution, in the format "dd/MM/yyyy" | | **Assignee** | (Full) name of the user who was assigned to the Test Run | | **Executed by** | (Full) name of the user who has executed the Test Run | | **Test Environment(s)** | Test Environment(s) of the Test Execution associated with the Test Run, delimited by a comma (e.g., "Firefox", "Edge, "Windows") | | **Started on** | Timestamp of when the Test Run started, in the format "dd/MM/yyyy hh:mm:ss" | | **Finished on** | Timestamp of when the Test Run finished, in the format "dd/MM/yyyy hh:mm:ss" | | **Elapsed** | Elapsed time, in the format "hh:mm:ss", for the Test Run; for data-driven tests, this corresponds to the overall Test Run elapsed time | | **Elapsed (sec)** | Elapsed time, in seconds, for the Test Run; for data-driven tests, this corresponds to the overall Test Run elapsed time | | **Comment** | Test Run comment | | **Total Defects** | Number of unique defects linked/reported to the Test Run, either globally, at the step level, including from iterations in data-driven Tests | | **Defects** | List of Issue keys of unique [Defects](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44566527) linked/reported to the Test Run, either globally, at step level, including from iterations in data-driven Tests, delimited by a comma | #### Test Runs Including Iterations Sheet This sheet presents: - A line for each Test Run that is non-data-driven (i.e., a standard run). - For data-driven Tests, a line for each iteration (i.e., data row) of the Test Run. Therefore, this sheet is like a superset of the *Test Runs Sheet*, where data-driven test runs are expanded to multiple rows. The fields displayed in this sheet are similar to the *Test Runs Sheet*. #### Input Issues Sheet This sheet lists the Issues that were used as input for generating the report. | **Column** | **Description** | | --- | --- | | **Key** | Input Issue's key | | **Issue Type** | The name of the Issue Type of the input Issue (e.g., "Test", "Test Execution") | | **Summary** | Summary field of the input Issue | | **Priority** | Priority field of the input issue | | **Component(s)** | Component field of the input issue, delimited by a comma (e.g., "core, "backend") | | **Priority** | Priority field of the input Issue (e.g., "Major") | | **Status** | Workflow status of the input Issue (e.g., "Reviewed") | | **Assignee** | The display name of the assignee of the input Issue | ## Customizing the Report ### Sections that Can be Hidden or Shown The report has some variables/flags that can be used to show or hide some sections whose logic is already implemented in the Template. These variables are defined at the top of each sheet, at the report template; the variables are scoped just to the current sheet. On the Template, use one of these values for flag type of variables: - **0**: to hide a section. - **1**: to show a section. The format for other types of variables is detailed below. > ℹ️ **Example of setting a variable to, in this case, render information on the sheet "Test Runs including iterations"** > ℹ️ > ℹ️ ${set(renderThisSheet, 1)} | **Variable/Flag** | **Purpose** | **Default** | **Example(s)** | | --- | --- | --- | --- | | **renderThisSheet** | - To render this sheet. - Format: 0 or 1. | 0 (shown) | `${set(renderThisSheet, 1)}` | | **statusesToInclude** | - To render Test Runs whose reported status is one in this list (delimited by a comma); use an empty string '' to include all statuses - Format: '," | ''
(i.e., all statuses) | `${set(statusesToInclude, 'FAILED')}`
`${set(statusesToInclude, 'FAILED,EXECUTING')}`
`${set(statusesToInclude, '')}` | ### Adding or Removing Information to/from the Report Since this report is column-based, if some columns are not relevant to you, you should be able to delete them. Make sure that no temporary variables are created in the cells of this column that are used in other subsequent columns. To add additional information, usually you're thinking of adding fields of the: - Test Run itself. - Test associated with the Test Run. - Test Execution associated with the Test Run. Eventually, also fields of the: - Test Plan(s) associated with the Test Execution that is associated with the Test Run. - Covered Issue(s) associated with the Test that is associated with the Test Run. The latter may be harder to implement, so we won't consider them here. #### Adding a Field from the Related Test Issue  Let's say you have a *Severity* field on the Test Issue that is connected to each Test Run, and that you want to show on the report. You can copy the column *Test Summary* and adapt it: 1. Insert column. 2. On the *Test Runs* sheet: 1. Copy *Test Summary* (i.e., insert a column  next to it and copy the values from the existing *Test Summary* column). 2. Change 1. `${TestRuns[n].Summary} to ${TestRuns[n].Severity}` 2. `${TestExecutions[j].TestRuns[n].Severity} to ${TestExecutions[j].TestRuns[n].Severity}` 3. Follow a similar approach for the *Test Runs including iterations* sheet. #### Add Total Evidence Count Let's say you want to know the total count of evidence items for each run and iteration. You can add the column next to the *Comment* one, then the code logic is similar to the Defect count. Please see the snippet below (this exact syntax is for the *Test Runs* tab): ##### **EvidenceSnippet** ```javascript ${set(totalStepEvidenceCount, 0)} ${set(totalEvidenceCount, 0)} #{if (%{!${TestExecutions[j].TestRuns[n].IsDataDriven}})} #{for m=TestExecutions[j].TestRuns[n].TestStepsCount} #{for l=TestExecutions[j].TestRuns[n].TestSteps[m].EvidencesCount} ${set(totalStepEvidenceCount,%{${totalStepEvidenceCount} + 1 })} #{end} #{end} ${set(totalEvidenceCount,%{${totalStepEvidenceCount} + ${TestExecutions[j].TestRuns[n].ExecutionEvidencesCount} })} #{end} #{if (%{${TestExecutions[j].TestRuns[n].IsDataDriven}})} #{for m=TestExecutions[j].TestRuns[n].IterationsCount} #{for k=TestExecutions[j].TestRuns[n].Iterations[m].TestStepsCount} #{for l=TestExecutions[j].TestRuns[n].Iterations[m].TestSteps[k].EvidencesCount} ${set(totalStepEvidenceCount,%{${totalStepEvidenceCount} + 1 })} #{end} #{end} ${set(totalEvidenceCount,%{${totalStepEvidenceCount} + ${TestExecutions[j].TestRuns[n].ExecutionEvidencesCount} })} #{end} #{end} ${totalEvidenceCount} ``` > ℹ️ - The core part of the path to the target (i.e. "TestExecutions[j].TestRuns[n]." above) is dependent on the source Issue Type, so you will need to adjust it based on the row in the Template. You can refer to other cells in the same row for the path syntax. > ℹ️ - For the *Test Runs including Iterations* tab, you will only need the content of one of the two "if" loops for each row and, for the data-driven rows, you will need to drop the additional "for" loop (i.e. "`#{for m=TestExecutions[j].TestRuns[n].IterationsCount}`" from the second "if" loop in the snippet above). # Performance Performance can be impacted by the information that is rendered and by how that information is collected/processed. The number of Test Runs, depending on scenarios, can be considerably high, especially with CI/CD. As this report sums up a lot of information, please use it wisely. Data-driven Tests may also add an overhead, as iterations need to be individually processed, for collecting all the reported/linked Defects for example. > ℹ️ **Tips** > ℹ️ > ℹ️ - Use the variables/flags to adjust sections or the Test Runs that will be processed/shown in the report; more info in *Customizing the Report* section. > ℹ️ - Limit the number of input Issues; in [Xporter](https://docs.getxporter.app/space/XPORTERCLOUD/), there's a global setting for this purpose. # Limitations - Test Runs comments are not formatted. - Gherkin scenario outlines are not considered data-driven (i.e., only one Test Run will appear).
Support/Troubleshooting If you have questions or technical issues, please [contact the Support team via the Customer Portal (Jira service management)](https://jira.getxray.app/servicedesk/customer/portal/2/user/login?destination=portal%2F2%2Fcreate%2F28) or [send us a message using the in-app chat](https://getxraydocs.atlassian.net/wiki/spaces/XRAYCLOUD/pages/44577312).