A Test Execution is an issue type that aggregates a user-determined collection of Tests. It monitors and verifies if those Tests are working as expected in a target context and environment. Through the Overall Execution Status, updated as each test is being performed, the user is informed about the progress of the Test Execution, including which tests passed, failed, are being executed or waiting to be performed.

An association between a Test Execution issue and a Test issue is called a Test Run. Hence, a Test Execution is composed of a list of Test Runs.

A Test Execution issue can be assigned to a Tester. It is also possible to assign individual Test Runs to several Testers. 

Sub Test Execution

 You can create a Test Execution as a Sub-Task in the Requirement Issue View Page. Learn more here.

Fields

 A Test Execution has the following custom fields:

  • Begin Date - the date the Test Execution is planned to start 
  • End Date - the date the Test Execution is planned to end 
  • Revision - an optional and open text field used to define the source code version, or the build number/name, of the product (e.g. SUT) targeted by testing; usually used in combination with the Fix Version field
  • Test Environments - environment (e.g. browser, device, testing stage) in which the Test Run is executed. Test Environments can be used in several ways depending on how you want to analyze results; more info here


Complementary, the Fix Version field should be defined as a means to identify that the Test Execution is going to be performed in the context of a given Release. By doing so, the Test Execution also becomes an artifact of that release, similar to other issues belonging to that release.


Version of the SUT

Whenever testing, it's important to clearly identify the exact version of the artifact where testing was performed on. This is done as a combination of the values for the Fix Version and the Revision fields.


The "version" of the system/application under test is usually characterized by the product version (i.e. the value of the Fix Version field), in case product versioning (i.e. defined in the project settings using Releases) is being used. Defining the Fix Version is crucial because it will affect the scope of the Release in Jira and also the coverage and related reports.

However, this is not enough to fully identify the product that was really tested.

During the development of a Release, code changes as commits are pushed to the source code versioning system (e.g. Git, SVN, CVS). This ultimately leads to new builds that can be targeted by testing.

Therefore, we can either identify the version of the source code, or the build produced by it, using the Revision field.


These custom fields are located in the native Jira sections. The Revision can be found in the Details section of the Test Execution issue while the Begin and End dates are located in the Dates section.

Sections

Xray contributes an additional section to the Test Execution view issue screen.

Tests Section

The Tests section provides the following views:

Overall Execution Status

This view has a colored progress bar where you can clearly see the status of a Test Execution. Final statuses float to the left side; the last status color to appear is the TODO status.

It is possible to filter tests by status in Test Execution Progress Bar as shown in the image below:

Test Run Filters

A filter section is also present above the Test Runs table to allow you to filter Test Runs  by Assignee, Status, Test Set, or open text.

Test Runs Table

The Tests section provides a table with all Test Runs created for the Test execution issue. Each row in this table represents a Test Run and contains a set of actions that can be performed over a Test Run.

The Columns table can be configured by you. It also provides bulk operations and drag-and-drop to change rank of Test Runs.

When Xray is installed, the default columns are: Rank, Key, Summary, Test Type, #Req, #Def, Test Sets, Assignee and Status where Rank and Status are fixed and cannot be removed or changed.

The #Req column provides the number of Requirements that are tested with each Test issue.

The #Def column provides the number of Defects created during the Test Execution or already related to the Test issue. For more information on how the calculation is done for this field, please refer to Defects Count.



Actions

NEW

Now it is possible to filter tests by clicking on "Overall Execution Status" in Test Execution Progress Bar

Create

To create a new Test Execution Issue, 

Step 1: Click the Create Issue at the top of the screen to open the Create Issue dialog box / page.

Step 2: Select the Project and on Issue Type, select Test Execution.

Step 3: Under the General tab, type a Summary for the test execution and fill all appropriate fields — mandatory fields are marked with an asterisk.

Step 4: Under the Test Execution Details tab, you can populate the Test Execution custom fields provided by Xray, namely, the Begin/End date and the Revision. In this tab, you can also associate Tests or Test Sets to create Test Runs.

Optional: To create a series of similar test executions – within the same Project – select the Create another checkbox at the bottom of the dialog.

Step 5: When you are satisfied with the content of your test execution, click the Create button.

Step 5a: If you selected the Create another checkbox, a new Create Issue dialog appears. This issue is automatically pre-populated with your previous test execution details, while leaving the Summary field blank.

Edit

The Edit Issue dialog box / page is similar to the Create Issue dialog box / page, and can be accessed through the following steps:

Step 1: Locate the Test Execution you want to edit.

Step 2: Click the Edit button (at the top-left of the 'view issue' page) to open the Edit Issue dialog box / page.
Alternatively, hover over a field and click the pencil icon to edit it inline. 

Step 3: Modify your Test Execution details in the appropriate fields using either the General or Test Execution Details tab of the Edit Issue view.

Step 4: Click the Update button to save your changes.

Clone

When cloning a Test Execution issue, all the Test issue associations will also be cloned.  However, a new Test Run is created for each Test so that the new Test Execution issue has no execution state. Also:

  • If the test execution contains links to other issue(s), you can select whether or not to include the links in the new clone test execution.
  • If the test execution contains sub-tasks, you can select whether or not to create the sub-tasks in the new clone test execution.
  • If the test execution contains attachments, you can select whether or not to include the attachments in the new clone test execution.

To clone an existing Test Execution issue:

Step 1: Open the Test Execution you wish to clone.

Step 2Select More > Clone. The Clone Issue screen will appear.

Step 3Edit the clone Test Set's Summary if you wish.

 Step 4: Click Create.

Adding Tests

Tests and Test Sets can be associated with Test Execution issues.

Add Tests

To add Test issues to a Test Execution:

Step 1: Open the Test Execution you wish to associate a Test with.

Step 2: Select Add > Tests  from the dropdown button in the Tests section.

Step 3: Open the Search tab.

How to use de Issue Picker Dialog

This is an example of how to add tests, using some of the features available in the Search tab.

For more information, please visit the Issue Picker Dialog detailed overview.

Step 4: Configure the search.

-Use the Filters dropdown button to include extra fields in the search criteria.

-Fill in the filters values.

-Press the search button.

Please note

Notice the setting for ignoring Tests in non-executable statuses at the bottom-left side. By default it will be selected, therefore restricting the shown Tests available for adding accordingly to the configuration "Miscellaneous > Workflow Status > ".


 

Step 5: Select and Add tests.

- Once the results are obtained you can order, add/remove columns to the results list (the columns selection is saved as a preference).

- Before adding the selected tests, you can define the execution assignee using the upper right corner Assignee dropdown.

Please note

Since Xray 3.4.0, the Assignee dropdown provides you a suggestions list with the user type fields found in the tests. 

If you choose the "Test Reporter" from the list, the assignee of the resultant Test Run will be the user set in the Test Reporter field of the respective test.

- You can also select a specific version of each Test to execute, provided that Tests have multiple active versions.

Please note

Since Xray 7.0, you can select the Test version to execute. You need to have Xray Enterprise installed.

- Check the tests you wish to add from the listed results. Here you can choose 


Step 6:  Add tests by pressing "Add selected".

Add Tests from Test Sets

In practical terms this operation is very similar to adding tests, instead you will just be searching and selecting Test Set issues.

Keep in mind that when adding a Test Set issue to a Test Execution, all the Test issues associated with the Test Set are added to the Test Execution. 

The Test Set issue is not directly associated with a Test Execution, so changes to the Test Set after it was added will not affect the Test Execution.


Associate with a Test Plan

You can associate Test Executions with Test Plans by editing the custom field Test Plan in the Test Execution issue view page, or directly in the create issue screen in the Test Execution Details tab.

If the Test Execution is associated with a Test Plan and contains Tests that are not in the Test Plan, a warning message will be displayed telling you how many tests in the Test Execution are not in the Test Plan with a button to add these tests to the Test Plan.

There will be one warning message for each Test Plan associated with the Test Execution that does not contain all the Tests in the Test Execution.

Remove Tests

 To remove Tests from the Test Execution (i.e. delete Test Runs):

Step 1: Open the Test Execution issue.

Step 2: Hover over the desired Test Run (under the Tests section on the view page)

Step 3: Click the Actions button that appears in the last column. The actions menu should pop up with the available actions.

Step 4: Click Remove action. A confirmation message appears.

Step 5: Choose Delete to delete the Test Run or Cancel to abort this action.

Assigning Work

There are two ways to distribute work between Testers.

  • A Test Execution issue can be created per Tester by assigning the issue to a specific Tester, or 
  • Within a Test Execution issue, you can assign individual Test Runs to Testers. Test Runs are assigned to the Test Execution issue assignee by default. Only users with permission to assign the Test Execution issue can assign Test Runs. 


When assigning individual Test Runs, it is also possible to assign based on any user type field present on the Test cases. For instance, you'll be able to assign Test Runs to the Test issue assignee. This allows users that are normally used to execute a specific Test case (that are assigned to the Test case), to be the default assignees on Test Runs within a Test Execution.


To assign individual Test Runs to testers,

Step 1: Open the Test Execution issue.

Step 2: Hover over the desired Test Run (under the Tests section on the view page)

Step 3: Click the Actions button that appears in the last column. The actions menu should pop up with the available actions.

Step 4: Click the Assign action. The assign dialog appears.

Step 5: Start typing to find the user to assign the Test Run and click Assign to complete the action or Cancel to abort it.

Change Test Run Rank

There are three ways to change the rank of Test Runs:

Drag-and-Drop

Step 1: Open the Test Execution issue.

Step 2: Hover over the desired Test Run row (under the Tests section on the view page) and the drag handle column.

Step 3: Click the drag handle and move the Test Run up or down in the current page.

Change Rank Action

This action allows users to change the Rank of a Test run between pages.

Step 1: Open the Test Execution issue.

Step 2: Hover over the desired Test Run (under the Tests section on the view page)

Step 3: Click the gear that appears in the last column. The actions menu should pop up with the available actions.

Step 4: Click the Move To action. The change rank dialog appears.

Step 5: Enter the desired index number (1-based) and click OK.

Apply Rank Button:

This button will apply the current display order to the Test Run Rank.

Step 1: Open the Test Execution issue.

Step 2: Click on the header cell of the sortable column you want to sort by.

Step 3: Click the Apply Rank button and confirm the action on the pop up.

Step 4: The Rank of the Test Runs is updated to match the ordering of the Test Runs by the selected column.

Step 5: The table reverts to sorting by Rank (ascending).





Execute Test

To execute a Test from a Test Execution:

Step 1: Open the Test Execution you wish to run.

Step 2: Hover over the desired Test Run (under the Tests section on the view page)

Step 3: Click the Execute button that appears in the last column. The actions menu should pop up with the available actions.

Step 4Click Run action to open the Execution page to start the execution.

You can see a detailed explanation of the execution screen here.


Note: If you does not have permission to execute the Test Run, the "Run" option will not be available.


Learn more

This action may trigger some email notification(s) depending on your Xray settings. For more info on email notifications and how to set-up them, please have a look at Email Notifications.

Execute Test Inline

To execute Test Runs inline from the Test Execution view screen, this option must be enabled in the Xray administration page. The option for setting the Test Run status manually without having to execute all Test Steps (or Examples, in the case of Cucumber Tests) must also be enabled in the Xray administration page.

Given that the above options are enabled and you have permission to execute the Test, the context menu for executing Test Runs, accessed by the Gear button on each Test, must show the available transitions. 

When executing inline Tests, the status of the manual steps (or Examples, in the case of Cucumber Tests) can be changed automatically. The following rules are applied:

  • PASS: All step statuses will be changed to PASS independently of their current status.
  • FAIL: All step statuses will remain unchanged, even if they are all PASS. This is because a Test might fail due to a Pre-Condition.
  • TODO: All step statuses will be changed to TODO.  
  • ABORTED: The Test Execution status will be changed to ABORTED. 
  • EXECUTING: The Test Execution status will be changed to EXECUTING.
  • BLOCKEDThe Test Execution status will be changed to BLOCKED.
  • CUSTOMThe Test Execution status will be changed to the chosen custom status.

To execute a Test Run inline:

Step 1: Open the Test Execution issue you wish to run.

Step 2: Click the Execute button located in the last column of the Test Runs table and select one of the available statuses.


Learn more

This action may trigger some email notification(s) depending on your Xray settings. For more info on email notifications and how to set-up them, please have a look at Email Notifications.

View Test Run Details

You can always view and modify the Test Run details after execution. If you do not have permission to execute the Test Run, you can still view the execution details in read-only mode, given that you have permission to browse the Test Execution issue.

To view the execution details from a Test Run:

Step 1: Open the Test Execution you wish to view the execution details.

Step 2: Hover over the desired Test Run (under the Tests section on the view page).

Step 3: Click the Execute button that appears in the last column. The actions menu should popup with the available actions.

Step 4: Click the Execution Details action to open the Execution page and view the details.

Bulk Operations

Xray provides the following bulk operations on the Test Runs table.  

  • Execute Inline - Executes inline all the selected Test Runs.
  • Assign - Assigns all the selected Test Runs to a particular user.
  • Remove - Deletes all the selected Test Runs.

To execute these bulk actions:

Step 1: Open the Test Execution issue.

Step 2: Select the desired Test Runs. This can be done by using the checkbox that appears on each Test Run row or, since v3.1, by using the Bulk dropdown button options for easy selecting all Test Runs or the ones in the current page.


Please note

As of v3.1, the selection will be preserved even if you navigate between pages.


Step 3: Click the bulk actions gear located on the actions column header and choose one of the available actions.



Learn more

Bulk operations may trigger some email notification(s) depending on your Xray settings. For more info on email notifications and how to set-up them, please have a look at Email Notifications.

Filter Test Runs

To filter the Test Runs table, you can pick any native or custom field that are available in the Global scope in your Test Issue Type. The 5 default fields are:

  • Test Set - show only Test Runs that are associated with a specific Test Set issue.
  • Test Run Assignee - show only Test Runs that are assigned to a specific user.
  • Execution Status - show only Test Runs with a specific status.
  • Component -  show only Tests that contain at least of the selected Components.
  • Contains Text -  show only Test Runs with Tests containing the entered text. This will query Test issues with the JeQL keyword text ~  ...


You can add or remove new fields by clicking at the More button. Note: The 5 default fields can't be removed.

You can also filter the tests by using any valid JQL.

Step 1: Open the Filter(s) dialog.

Step 2: Select the Advanced tab.

Step 3: Write any valid JQL in the field.

Step 4: Click Apply

There will also be an JQL AutoComplete suggestions to help you to write your JQL query.

Only the Tests are returned by the JQL will be displayed in the Test Run table.


Since v3.1, the filters will be stored in the user and Test Execution context, and will be applied whenever going into the execution screen from one of the filtered Tests.


The execution screen shows a small notice, so you can quickly identify that the filter has been applied thus affecting the navigation options (Previous/Next) available in the execution screen.



If you decide to come back to the Test Execution issue screen, the filter will remain. If you wish, you can clear the filter in order to see all Test Runs once again. You can do this by clicking on the Clear button and then press Apply.

Configure Test Run Columns

More info in Test Runs.

Order Test Runs

Since v3.4.0, it is now possible to sort test runs by an Issue Field of the Test. Any sortable JIRA field and Custom Field is eligible. Simply add it to the table using the Columns menu and then click it's header cell on the table to sort by that field. Click again to toggle between ascending and descending order. Just as the configured columns a user has selected are stored, the selected order is also stored.


Export to Cucumber

Cucumber Test Executions in JIRA can be exported to feature files ready to be executed in Cucumber. The generated feature files follow the rules defined in Export Rules and take into account the context of the Jira Test Execution issue.

A single Cucumber feature file should be generated and prompted to be downloaded for the Cucumber Test contained on the Test Execution, unless the target Test Execution contains multiple Cucumber Tests or the Cucumber Test is associated with multiple Requirements. In this case, a FeatureBundle zip file will be generated containing a single Cucumber feature file for every Requirement associated with the target Cucumber Test contained on the Test Execution.

To export a Test Execution issue containing Cucumber Tests to feature files, 

Step 1Open the Test Execution issue view page. 

Step 2: Select More > Export to Cucumber. The file will be downloaded from the browser.


Export Test Runs to CSV

Xray Test Runs can be exported to CSV. The Xray export action will export to CSV the most relevant fields for Test Runs such as:

 

  • Test Key
  • Test Execution Key
  • Executed By
  • Assignee
  • Start
  • Finish
  • Defects issue keys (list separated by ",")
  • Attachment Jira links (list separated by ",")
  • Comment
  • Status

 

To export a Test Run to CSV,  

Step 1Open the Test Execution issue view page.  

Step 2: Select More > Export Test Runs to CSV. The resulting CSV file will be downloaded to the browser.

 


Import Execution Test Results

Importing from Test Execution view issue screen 

Automated Test results can be imported back to Jira using the Test Execution view issue screen. The Test results can be imported from JSON/XML external files that must follow an Import Results specific scheme.

When importing results, only the Tests contained in the interception between the Tests in the JSON/XML file and the Tests in the selected Test Execution are updated, while the remaining, if any, are ignored. After importing, a new entry with the result of the operation is registered in the Activity Log of the Test Run.

To import Automated Test results to a Test Execution issue containing automated Tests, 

Step 1Open the Test Execution issue view page. 

Step 2: Select More > Import Execution Results. The Import Automated Test Results dialog will be prompted.

Step 3: Click the Browse button to navigate to the local disk and select the JSON/XML/ZIP file with the automated execution results for the Test Execution. You can see the support file schemas here.

Step 4: Click Import.

Importing from the Execution Details Page

You can also import the Automated Test results from the Execution Details page:

Step1: From Test Execution view issue screen, go to the Execution Details page by clicking Execute (or by

clicking Execution Details if the test has been previously executed).


Step 2: Click the Import Execution Results button.

Step 3: Click the Browse button to navigate the local disk and select the JSON/XML/ZIP file with the automated execution results for the Test Execution.

Step 4: Click Import.



Now it is possible to filter tests by clicking on "Overall Execution Status" in Test Plans Progress Bar.


Now it is possible to view the associated Test Plans of a given Test in the Execution Details view page as shown below.
Second image shows behaviour if more than 1 test plan is associated with the Test


Single Test Plan associated with test

Multiple Test Plan associated with test