Purpose

This report enables you to extract details of a Test Plan, such as the Tests that are part of it, Defects, Requirements and Test Executions, so that you can generate a document report focusing in what matter the most for your team, or even share it with someone else that hasn't access to Jira.

Possible usage scenarios:

  • see all the requirements covered by the Test Plan
  • see all the defects linked to this Test Plan
  • see an overall status summary of the Test Plan
  • see a summary of the Tests that are part of the Test Plan
  • check a specific detail of a Test Execution (like evidences, attachments, assignee, etc)

Output Example(s)

The following screenshots shows an example of the sections you should expect in this report.






How to use


This report can be generated from the Issue details screen.


Learn more

General information about all the existing places available to export from, and how to perform it, is available in the Exporting page.

Source data

This report is applicable to:

  • 1 Test Plan issue

Output format

The standard output format is .DOCX, so you can open it in Microsoft Word, Google Docs, and other tools compatible with this format. 

Report assumptions

The template has a set of assumptions that you make sure that your Jira/Xray environment complies with:

  1. Issue types having these names
    1. "Test", "Test Plan", "Test Execution"

If any of these assumptions is not met, you need to update the template accordingly.

Usage examples

Export all details obtained in the context of a given Test Plan

  1. open the Test Plan issue and export it using this template

Understanding the report

The report shows detailed information about the Test Plan provided.

Layout

The report is composed by several sections. Two major sections are available: Introduction and Test Executions details.

By default, and to avoid overload/redundancy of information, only the "Introduction" section will be rendered; you can change this behavior on the template (more info ahead).

"Introduction" section

This section is divided into 6 sub-sections to have an overview about the Test Plan we have just exported:

  1. Document Overview
  2. Test Plan Details
  3. Requirements covered by the Tests in this Test Plan
  4. Overall Execution Status
  5. Defects
  6. Tests Summary

Each of these sections is explained below.

Document Overview

Brief description of what you will find in this report and how it was generated.

Test Plan Details

In this section we are extracting the Test Plan key in the header and show the Begin and End Date (formatted as demonstrated below), the Summary and the Description present in the Test Plan.

FieldDescriptionSample Code
Begin DateTimestamp of the Begin Date field present in the Test Plan (with proper format)${dateformat("dd-MM-yyyy HH:mm:ss"):Begin Date}
End DateTimestamp of the End Date field present in the Test Plan (with proper format)${dateformat("dd-MM-yyyy HH:mm:ss"):End Date}
SummarySummary of the Test Plan${Summary}
DescriptionDescription of the Test Plan (as this field accepts wiki markup we will use "wiki:" in the code to be interpreted by the document)${wiki:Description}


The output will have the following information, notice that as the Description field support wiki markup we are using "wiki:" keyword so that it is correctly interpreted.

Requirements covered by the Tests in this Test Plan

In this section we have an overview of all the requirements that are covered by Tests in this Test Plan, we extract the Key, Summary and the Workflow Status removing all the repeated entries.

FieldDescriptionSample Code
KeyKey of the requirement (in this case we are adding it as link)@{title=${Tests[n].Links[k].Key}|href=${BaseURL}/browse/${Tests[n].Links[k].Key}
SummarySummary of the requirement${Tests[n].Links[k].Summary}
Workflow StatusWorkflow Status of the requirement${Tests[n].Links[k].Status}


The requirements are listed in a table with the informations explained above.


Overall Execution Status

As the name suggests we have an overview about the executions of the Tests in this Test Plan, here you will have information about how many Tests you have in this Test Plan and what are the statuses of their executions.

To obtain this information we are using:

FieldDescriptionSample Code
TestsCountThe total number of Tests in this Test Plan${TestsCount}
#Tests

To extract the count of the overall execution status per each <status> (TO DO, EXECUTING, PASSED, FAILED, ABORTED)

${Overall Execution Status.<status>.Count}
Percentage

To extract the percentage of the overall execution status per each <status> (TO DO, EXECUTING, PASSED, FAILED, ABORTED)

${Overall Execution Status.<status>.Percentage}


This will produce the following output:


Defects

In this section we are listing all the defects found that are associated with this Test Plan, we consider defects associated with TestRuns, defects in Test Steps or defects found during the iterations. We do not print duplicates.

FieldDescriptionSample Code
KeyKey of the Defect
  • TestRun
    • @{title=${TestExecutions[n].TestRuns[a].ExecutionDefects[d].Key}|href=${BaseURL}/browse/${TestExecutions[n].TestRuns[a].ExecutionDefects[d].Key}}
  • TestSteps
    • @{title=${TestExecutions[n].TestRuns[a].TestSteps[j].Defects[m].Key}|href=${BaseURL}/browse/${TestExecutions[n].TestRuns[a].TestSteps[j].Defects[m].Key}}
  • Iteration TestSteps
    • @{title=${TestExecutions[n].TestRuns[a].Iterations[it].TestSteps[r].Defects[dc].Key}|href=${BaseURL}/browse/${TestExecutions[n].TestRuns[a].Iterations[it].TestSteps[r].Defects[dc].Key}}

Summary

Summary of the Defect
  • TestRun
    • ${TestExecutions[n].TestRuns[a].ExecutionDefects[d].Summary}
  • TestSteps
    • ${TestExecutions[n].TestRuns[a].TestSteps[j].Defects[m].Summary}
  • Iteration TestSteps
    • ${TestExecutions[n].TestRuns[a].Iterations[it].TestSteps[r].Defects[dc].Summary}
PriorityPriority of the Defect
  • TestRun
    • ${TestExecutions[n].TestRuns[a].ExecutionDefects[d].Priority}
  • TestSteps
    • ${TestExecutions[n].TestRuns[a].TestSteps[j].Defects[m].Priority}
  • Iteration TestSteps
    • ${TestExecutions[n].TestRuns[a].Iterations[it].TestSteps[r].Defects[dc].Priority}


The Defects appear in the document as a table with information regarding the defects found during the executions of the Test Plan.


Tests Summary

In this section we have a table with information regarding the Tests included in this Test Plan. You can find the following information about each Test:

FieldDescriptionSample Code
KeyKey of the Test in a link form@{title=${Tests[n].Key}|href=${BaseURL}/browse/${Tests[n].Key}}
SummaryThe Summary of the Test${Tests[n].Summary}
Issue AssigneeFull name of the assignee${fullname:Tests[n].AssigneeId}
RequirementsList of requirements covered by this Test (Check the template to see the extra cycle we need to list this information)${Tests[n].Links[k].Key}
Test ExecutionsList of ids of the Test Executions for each Test (Check the template to see the extra cycle we need to list this information)${Tests[n].TestExecutions[j].Key}
Latest StatusLatest Status of the execution${Tests[n].TestStatus}


This information is presented in a table as we can see below:


Some particularities to highlight a different behavior about the code needed to show the Tests Runs section:

  • Ability to put the Tests with a particular status on top of the table (more info ahead).
  • Usage of ${fullname:Tests[n].AssigneeId}, this allows us to fetch the full name of the assignee instead of the key associated to it.


Test Executions

This section will gather all the information related to each Test Execution of each Test in the Test Plan with all the possible details.

It is composed with several sub-sections that will be filled with information if it is available or be filled with a message showing that no information is available.

Test Executions Summary

This section have a table with information regarding each Test Execution in this Test Plan (and will repeat these sections for each Test Execution). The information is presented as a table with the following fields:

FieldDescriptionSample Code
Execution statusExecution Status of the Test Run${TestExecutions[n].TestRuns[a].Execution Status}
AssigneeFull Name of the Assignee of the Test Run${fullname:TestExecutions[n].TestRuns[a].AssigneeId}
Executed ByFull Name of the entity that has executed this Test Run${fullname:TestExecutions[n].TestRuns[a].Executed By}
Started OnTimestamp of the Started Date from the TestRun${dateformat('dd-MM-yyyy HH:mm:ss'):TestExecutions[n].TestRuns[a].Started On}
Finished OnTimestamp of the Finished Date from the TestRun${dateformat('dd-MM-yyyy HH:mm:ss'):TestExecutions[n].TestRuns[a].Finished On}


All of these fields have code to handle empty fields. The resulting table look like the one below.


Execution Defects

If any Defects was found and associated globally with a TesRun it will appear here in the form of a table with the following fields:

KeyDescriptionSample Code
KeyJira Key of the Defect in the form of a link@{title=${TestExecutions[n].TestRuns[a].ExecutionDefects[d].Key}|href=${BaseURL}/browse/${TestExecutions[n].TestRuns[a].ExecutionDefects[d].Key}}
SummarySummary of the Defect${TestExecutions[n].TestRuns[a].ExecutionDefects[d].Summary}
PriorityPriority associated with the defect${TestExecutions[n].TestRuns[a].ExecutionDefects[d].Priority}


The table will be similar to the one below.

Execution Evidences

If any Evidence was attached to the TestRun we are showing it in table with the FileName and a screenshot if the Evidence is an image otherwise just a link.

To obtain that information we have used the following code:

KeyDescriptionSample Code
File NameThe File Name of the Evidence attached to the Execution@{title=${TestExecutions[n].TestRuns[a].ExecutionEvidences[d].Name}|href=${TestExecutions[n].TestRuns[a].ExecutionEvidences[d].FileURL}}
EvidenceThe Evidence attached to the Execution!{${TestExecutions[n].TestRuns[a].ExecutionEvidences[d].Evidence|maxwidth=100}}


The table in case of an Evidence is of the type image will have the following aspect:

Comment

The comment associated to the TestRun (${TestExecutions[n].TestRuns[a].Comment}).

Test Description

The description of the TestRun (${wiki:TestExecutions[n].TestRuns[a].Description}).

Test Issue Attachments

This section only appears if you have any attachments associated to the Test.

KeyDescriptionSample Code
File NameFile Name of the Attachment@{title=${TestExecutions[n].Tests[t].Attachments[b].Name}|href=${TestExecutions[n].Tests[t].Attachments[b].FileURL}}
AuthorThe Author of the attachment${fullname:TestExecutions[n].Tests[t].Attachments[b].Author}
File SizeFile Size of the attachments in bytes.${TestExecutions[n].Tests[t].Attachments[b].Size}


This appears in the document in a table form:

Preconditions

This section only appear if you have a Precondition associated with the TestRun.

KeyDescriptionSample Code
PreconditionThe definition of the precondition associated to the TesRun${wiki:TestExecutions[n].TestRuns[a].PreConditions[l].PreCondition.Definition}


A sub section will appear with the preconditions definitions.

Parameters

This section lists the existing parameters of the TestRun (we are iterating through the Parameters of the TestRun with: #{for m=TestExecutions[n].TestRuns[a].ParametersCount}).

KeyDescriptionSample Code
NameKey of the parameter

${TestExecutions[n].TestRuns[a].Parameters[m].Key}

ValueValue of the parameter${TestExecutions[n].TestRuns[a].Parameters[m].Value}


It will list the Key and the Value of each parameter in a table.

Iterations

This section uses a sentence to show how many interactions we will go into more details in the next sections.

KeyDescriptionSample Code
IterationsThe iterations count of the Test Run${TestExecutions[n].TestRuns[a].IterationsCount}


A sentence is added to the document with this information.

Iteration Overall Execution Status

To obtain the overall execution status of the iteration we use two variables:

KeyDescriptionSample Code
List of StatusesShow the List of Statuses${TestExecutions[n].TestRuns[a].IterationsOverallExecutionStatus}

TO DO

EXECUTING

PASS

FAIL

ABORTED

Overall Execution Status per Status

${TestExecutions[n].TestRuns[a].IterationsOverallExecutionStatus.TO DO}

${TestExecutions[n].TestRuns[a].IterationsOverallExecutionStatus.EXECUTING}

${TestExecutions[n].TestRuns[a].IterationsOverallExecutionStatus.PASSED}

${TestExecutions[n].TestRuns[a].IterationsOverallExecutionStatus.FAILED}

${TestExecutions[n].TestRuns[a].IterationsOverallExecutionStatus.ABORTED}


The above code will produce the below table.


Test Run details

In this section we are showing the Test Run details with the Name, Status and Parameters.

We extract that information using the following fields:

KeyDescriptionSample Code
Iteration NameName of the iteration${TestExecutions[n].TestRuns[a].Iterations[m].Name}
StatusStatus of the iteration

${TestExecutions[n].TestRuns[a].Iterations[m].Status}

Total ParametersTotal number of parameters

${TestExecutions[n].TestRuns[a].Iterations[m].ParametersCount}

ParametersLists all parameters in the form of Key=Value${TestExecutions[n].TestRuns[a].Iterations[m].Parameters}


This section will have the below appearance:

Iteration precondition definition

If a precondition is present we will use the following fields to extract that information:

KeyDescriptionSample Code
KeyIteration precondition key${TestExecutions[n].TestRuns[a].Iterations[m].PreConditions[l].Key}
DefinitionIteration precondition definition

${wiki:TestExecutions[n].TestRuns[a].Iterations[m].PreConditions[l].PreCondition.Definition}


This will produce an entry like the one below:


Iteration parameters details

For that given Iteration we are listing the parameters used, that information is extracted with the following fields:

KeyDescriptionSample Code
NameParameter Key${TestExecutions[n].TestRuns[a].Iterations[m].Parameters[l].Key}
ValueParameter Value${TestExecutions[n].TestRuns[a].Iterations[m].Parameters[l].Value}


It generates a table of the following form:

Iteration Test Step Details

In this section we are listing the details of an iteration, we are listing each step present with details, the code we use for that purpose is present in the below table.

KeyDescriptionSample Code

Step

The Step Number ${TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].StepNumber}
ActionAction defined in the Test Step${wiki:TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].Action}
DataData defined in the Test Step${TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].Data}
Expected ResultExpected Result defined in the Test Step${wiki:TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].ExpectedResult}
AttachmentsAttachments present in each Test Step (showing the FileURL and a screenshot in case of the Attachment being an image)

@{title=${TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].Attachments[sa].Name}|href=${TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].Attachments[sa].FileURL}}

!{${TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].Attachments[sa].Attachment|maxwidth=100}}

CommentComment${wiki:TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].Comment}
DefectsDefects associated to this Iteration

@{title=${TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].Defects[dc].Key}|href=${BaseURL}/browse/${TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].Defects[dc].Key}}

EvidenceFileURL and screenshot (if it is an image) of the Evidence

@{title=${TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].Evidences[e].Name}|href=${TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].Evidences[e].FileURL}}

!{${TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].Evidences[e].Evidence|maxwidth=100}}
StatusTest Step Status

${TestExecutions[n].TestRuns[a].Iterations[m].TestSteps[r].Status}


The above information is gathered in a table like the one below:

Test Details

This section shows the Test details, for that we are considering the different possible Test we can have in Xray: Generic, Manual and Cucumber. For each type we will fetch different information.

It may seem similar with the Iteration Test Step Details section but in this section we will show the Test details (not instantiated in each Iteration like the previous section).


TypeKeyDescriptionSample CodeOutput
GenericTest TypeTest Type field

${TestExecutions[n].TestRuns[a].TestType}

SpecificationDefinition of the Generic test${TestExecutions[n].TestRuns[a].Generic Test Definition}
CucumberTest TypeTest Type field

${TestExecutions[n].TestRuns[a].TestType}

Gherkin SpecificationGherkin specification of the Test${TestExecutions[n].TestRuns[a].Cucumber Scenario}
ManualStepStep Number${TestExecutions[n].TestRuns[a].TestSteps[r].StepNumber}

ActionAction of the Test Step${TestExecutions[n].TestRuns[a].TestSteps[r].Action}
DataData of the Test Step${TestExecutions[n].TestRuns[a].TestSteps[r].Data}
Expected ResultExpected Result of the Test Step${TestExecutions[n].TestRuns[a].TestSteps[r].ExpectedResult}
AttachmentAttachment of the Test Step

${TestExecutions[n].TestRuns[a].TestSteps[r].Attachments[sa].FileURL}

!${TestExecutions[n].TestRuns[a].TestSteps[r]. Attachments[sa]. Attachment|maxwidth=100}

CommentComment of the Test Step${wiki:TestExecutions[n].TestRuns[a].TestSteps[r].Comment}
DefectsDefects associated with the Test Step

${TestExecutions[n].TestRuns[a].TestSteps[r].Defects[dc].Key}

EvidenceEvidence with the Test Step

${TestExecutions[n].TestRuns[a].TestSteps[r].Evidences[e].FileURL}}

!${TestExecutions[n].TestRuns[a].TestSteps[r].Evidences[e].Evidence|maxwidth=100}
StatusStatus of the Test Step

${TestExecutions[n].TestRuns[a].TestSteps[r].Status}


Requirements linked with this test

For each Test we are listing the Requirements linked 

KeyDescriptionSample Code
Requirement KeyKey of the Requirement${TestExecutions[n].Tests[t].Links[j].Key}
Requirement SummarySummary of the Requirement${TestExecutions[n].Tests[t].Links[j].Summary}
Workflow StatusWorkflow status of the Requirement${TestExecutions[n].Tests[t].Links[j].Status}


This section present a table with that information like the one below:

Appendix A: Approval

This section is added for the cases where you need to have a signature validating the document.

Customizing the report

Sections that can be hidden or shown

The report has some variables/flags that can be used to show or hide some sections whose logic is already implemented in the template.

These variables are defined at the top of each sheet, at the report template; the variables are scoped just to the current sheet.

On the template, use one of these values for flag type of variables:

  • 0: to hide a section
  • 1: to show a section

The format for other types of variables is detailed ahead.


Example of setting a variable to, in this case, render information on the section "Test Executions"

${set(showTestRunDetails, 1)}

Variable/flagPurposedefaultexample(s)

showTestRunDetails

render the details section

  • format: 0 or 1

0

${set(showTestRunDetails, 0)}

showTestRunEvidences

render this section

  • format: 0 or 1

(Make sure to define showTestRunDetails at 1)

0

${set(showTestRunEvidences, 0)}

showTestRunAttachments

render this section

  • format: 0 or 1

(Make sure to define showTestRunDetails at 1)

0

${set(showTestRunAttachments, 0)}

showTestRunIterations

render this section

  • format: 0 or 1

(Make sure to define showTestRunDetails at 1)

0

${set(showTestRunIterations, 0)}

statusesToShowFirst

render Test Summary section whose reported status is one in this list first (delimited by comma); use an empty string '' to include all statuses

  • format: '<status1>,<status2>"


''

(i.e., all statuses)

${set(statusesToInclude, 'FAILED')}

${set(statusesToInclude, 'FAIL,EXECUTING')}

${set(statusesToInclude, '')}

Adding or removing information to/from the report

As this report is a document with different sections, if some sections are not relevant to you, you should be able to simply delete them. Make sure that no temporary variables are created in that section that are used in other subsequent sections or if any all conditional blocks are properly closed.

To add additional information, usually we're thinking of adding:

  • fields of the Test Plan itself
  • fields of the Tests associated with the Test Plan
  • fields of the Test Executions associated with the Test Plan

Eventually, also:

  • fields of the Test Runs(s)
  • fields of the covered issue(s) associated with the Test that is associated with the Test Plan

The later may be harder to implement, so we won't consider them here.

Exercise: add a field from the related Test issue 

Let's say we have a "Severity" field on the Test issue that is connected to the Test Plan, and that we want to show it on the report.

We can copy the column "Summary" from the "Tests Summary" section and adapt it.

  1. insert new column in the table
  2. on the "Tests Summary" section,
    1. copy "Summary" (i.e., insert a column  next to it and copy the values from the existing "Summary" column)
    2. change
      1. ${Tests[n].Summary} to ${Tests[n].Severity}

Performance

Performance can be impacted by the information that is rendered and by how that information is collected/processed.

The number of Test Runs and Tests depending on scenarios, can be considerably high, especially with CI/CD. As this report sum-up quite information, please use it wisely.

Data-driven tests may also add an overhead, as iterations need to be individually processed, for collecting all the reported/linked defects for example.


Some tips

  • Use the variables/flags to adjust sections or the Test Plan that will be processed/shown in the report; more info in "Customizing the report"
  • limit the number of input issues; in Xporter there's a global setting for this purpose

Known limitations

  • Test Plan comments are not formatted
  • Gherkin Scenario Outlines are not considered as data-driven (i.e., only one Test Run will appear)





  • No labels