A Test Execution is an issue type that aggregates a user-determined collection of Tests. It monitors and verifies if those Tests are working as expected in a target context and environment. Through the Overall Execution Status, updated as each test is being performed, the user is informed about the progress of the Test Execution, including which tests passed, failed, are being executed or waiting to be performed.
An association between a Test Execution issue and a Test issue is called a Test Run. Hence, a Test Execution is composed of a list of Test Runs.
A Test Execution issue can be assigned to a Tester. It is also possible to assign individual Test Runs to several Testers.
You can create a Test Execution as a Sub-Task in the Requirement Issue View Page. Learn more here. |
A Test Execution has the following custom fields:
These custom fields are located in the native Jira sections. The revision can be found in the Details section of the Test Execution issue while the Begin and End dates are located in the Dates section.
|
Xray contributes an additional section to the Test Execution view issue screen.
The Tests section provides the following views:
This view has a colored progress bar where you can clearly see the status of a Test Execution. Final statuses float to the left side; the last status color to appear is the TODO status.
A filter section is also present above the Test Runs table to allow you to filter Test Runs by Assignee, Status, Test Set, or open text.
The Tests section provides a table with all Test Runs created for the Test execution issue. Each row in this table represents a Test Run and contains a set of actions that can be performed over a Test Run.
The Columns table can be configured by you. It also provides bulk operations and drag-and-drop to change rank of Test Runs.
When Xray is installed, the default columns are: Rank, Key, Summary, Test Type, #Req, #Def, Test Sets, Assignee and Status where Rank and Status are fixed and cannot be removed or changed.
The #Req column provides the number of Requirements that are tested with each Test issue.
The #Def column provides the number of Defects created during the Test Execution or already related to the Test issue. For more information on how the calculation is done for this field, please refer to Defects Count.
To create a new Test Execution Issue,
Step 1: Click the Create Issue at the top of the screen to open the Create Issue dialog box / page.
Step 2: Select the Project and on Issue Type, select Test Execution.
Step 3: Under the General tab, type a Summary for the test execution and fill all appropriate fields — mandatory fields are marked with an asterisk.
Step 4: Under the Test Execution Details tab, you can populate the Test Execution custom fields provided by Xray, namely, the Begin/End date and the Revision. In this tab, you can also associate Tests or Test Sets to create Test Runs.
Optional: To create a series of similar test executions – within the same Project – select the Create another checkbox at the bottom of the dialog.
Step 5: When you are satisfied with the content of your test execution, click the Create button.
Step 5a: If you selected the Create another checkbox, a new Create Issue dialog appears. This issue is automatically pre-populated with your previous test execution details, while leaving the Summary field blank.
The Edit Issue dialog box / page is similar to the Create Issue dialog box / page, and can be accessed through the following steps:
Step 1: Locate the Test Execution you want to edit.
Step 2: Click the Edit button (at the top-left of the 'view issue' page) to open the Edit Issue dialog box / page.
Alternatively, hover over a field and click the pencil icon to edit it inline.
Step 3: Modify your Test Execution details in the appropriate fields using either the General or Test Execution Details tab of the Edit Issue view.
Step 4: Click the Update button to save your changes.
When cloning a Test Execution issue, all the Test issue associations will also be cloned. However, a new Test Run is created for each Test so that the new Test Execution issue has no execution state. Also:
To clone an existing Test Execution issue:
Step 1: Open the Test Execution you wish to clone.
Step 2: Select More > Clone. The Clone Issue screen will appear.
Step 3: Edit the clone Test Set's Summary if you wish.
Step 4: Click Create.
Tests and Test Sets can be associated with Test Execution issues.
To add Test issues to a Test Execution:
Step 1: Open the Test Execution you wish to associate a Test with.
Step 2: Select Add > Tests (under the Tests tab on the view page) to open the Add Tests dialog box.
Step 3: Select or search the Test issues to be associated with the Test Execution:
Select Test Issues - In order to select the Tests to be associated, you can, under the Select tab:
- Input the desired Test Issue Key on the Tests field.
- Click on the Down Arrow on the Tests field and select the Test from its History Search list.
- Click on the + button wich will prompt an Issue Selector Pop-Up.
- Click on the Search tab to use the Find Jira issues browser.
Search Test Issues - In order to search for Tests to be associated, you can, under the Search tab:
- Browse for the desired Test Project, Type or terms it Contains.
- Click on the More button to add more browse fields such as Label, Test Set, Test Set Fix Version, Requirement, Requirement Fix Version, Saved Filter and Workflow Status.
- Click on the Magnifier button after filling out the browse fields to get a list of matching entries
- Click on the Advanced button to perform a JQL search.
Step 4: Click Add.
When adding a Test Set issue with a Test Execution, all the Test issues associated with the Test Set are added to the Test Execution. The Test Set issue is not directly associated with a Test Execution, so changes to the Test Set after it was added will not affect the Test Execution.
Step 1: Open the Test Execution you wish to associate a Test Set with.
Step 2: Select Add > Tests (under the Tests tab on the view page) to open the Add Test Sets dialog box.
Step 3: Select or search the Test Set issues to be associated with the Test Execution.
Select Test Set Issues - In order to select the Test Sets to be associated, you can, under the Select tab:
- Input the desired Test Set Issue Key on the Test Sets field.
- Click on the Down Arrow on the Test Sets field and select the Test Set from its History Search list.
- Click on the + button wich will prompt an Issue Selector Pop-Up.
- Click on the Search tab to use the Find Jira issues browser.
Search Test Sets Issues - In order to search for Test Sets to be associated, you can, under the Search tab:
- Browse for the desired Test Set Project or terms it Contains.
- Click on the More button to add more browse fields such as Label, Fix Version, Saved Filter and Workflow Status.
- Click on the Magnifier button after filling out the browse fields to get a list of matching entries.
- Click on the Advanced button to perform a JQL search.
Step 4: Click Add.
You can associate Test Executions with Test Plans by editing the custom field Test Plan in the Test Execution issue view page, or directly in the create issue screen in the Test Execution Details tab.
If the Test Execution is associated with a Test Plan and contains Tests that are not in the Test Plan, a warning message will be displayed telling you how many tests in the Test Execution are not in the Test Plan with a button to add these tests to the Test Plan.
There will be one warning message for each Test Plan associated with the Test Execution that does not contain all the Tests in the Test Execution.
|
There are two ways to distribute work between Testers.
|
|
To execute a Test from a Test Execution:
Step 1: Open the Test Execution you wish to run.
Step 2: Hover over the desired Test Run (under the Tests section on the view page)
Step 3: Click the Execute button that appears in the last column. The actions menu should pop up with the available actions.
Step 4: Click Run action to open the Execution page to start the execution.
You can see a detailed explanation of the execution screen here.
Note: If you does not have permission to execute the Test Run, the "Run" option will not be available.
To execute Test Runs inline from the Test Execution view screen, this option must be enabled in the Xray administration page. The option for setting the Test Run status manually without having to execute all Test Steps (or Examples, in the case of Cucumber Tests) must also be enabled in the Xray administration page.
Given that the above options are enabled and you have permission to execute the Test, the context menu for executing Test Runs, accessed by the Gear button on each Test, must show the available transitions.
When executing inline Tests, the status of the manual steps (or Examples, in the case of Cucumber Tests) can be changed automatically. The following rules are applied:
To execute a Test Run inline:
Step 1: Open the Test Execution issue you wish to run.
Step 2: Click the Execute button located in the last column of the Test Runs table and select one of the available statuses.
You can always view and modify the Test Run details after execution. If you do not have permission to execute the Test Run, you can still view the execution details in read-only mode, given that you have permission to browse the Test Execution issue.
To view the execution details from a Test Run:
Step 1: Open the Test Execution you wish to view the execution details.
Step 2: Hover over the desired Test Run (under the Tests section on the view page).
Step 3: Click the Execute button that appears in the last column. The actions menu should popup with the available actions.
Step 4: Click the Execution Details action to open the Execution page and view the details.
Xray provides the following bulk operations on the Test Runs table.
To execute these bulk actions:
Step 1: Open the Test Execution issue.
Step 2: Click on the Bulk button to activate the bulk edit mode (under the Tests section on the view page). A checkbox appears on each Test Run row. The bulk actions button appears in the actions column header.
Step 3: Select the desired Test Runs.
Step 4: Click the bulk actions gear located on the actions column header and choose one of the available actions.
To filter the Test Runs table, select the desired filters in the filters section above the Test Runs table.
|
Cucumber Test Executions in JIRA can be exported to feature files ready to be executed in Cucumber. The generated feature files follow the rules defined in Export Rules and take into account the context of the Jira Test Execution issue.
A single Cucumber feature file should be generated and prompted to be downloaded for the Cucumber Test contained on the Test Execution, unless the target Test Execution contains multiple Cucumber Tests or the Cucumber Test is associated with multiple Requirements. In this case, a FeatureBundle zip file will be generated containing a single Cucumber feature file for every Requirement associated with the target Cucumber Test contained on the Test Execution.
To export a Test Execution issue containing Cucumber Tests to feature files,
Step 1: Open the Test Execution issue view page.
Step 2: Select More > Export to Cucumber. The file will be downloaded from the browser.
Xray Test Runs can be exported to CSV. The Xray export action will export to CSV the most relevant fields for Test Runs such as:
To export a Test Run to CSV,
Step 1: Open the Test Execution issue view page.
Step 2: Select More > Export Test Runs to CSV. The resulting CSV file will be downloaded to the browser.
Automated Test results can be imported back to Jira using the Test Execution view issue screen. The Test results can be imported from JSON/XML external files that must follow an Import Results specific scheme.
When importing results, only the Tests contained in the interception between the Tests in the JSON/XML file and the Tests in the selected Test Execution are updated, while the remaining, if any, are ignored. After importing, a new entry with the result of the operation is registered in the Activity Log of the Test Run.
To import Automated Test results to a Test Execution issue containing automated Tests,
Step 1: Open the Test Execution issue view page.
Step 2: Select More > Import Automated Results. The Import Automated Test Results dialog will be prompted.
Step 3: Click the Browse button to navigate to the local disk and select the JSON/XML/ZIP file with the automated execution results for the Test Execution.
Step 4: Click Import.
You can also import the Automated Test results from the Execution Details page:
Step1: From Test Execution view issue screen, go to the Execution Details page by clicking Execute (or by
clicking Execution Details if the test has been previously executed).
Step 2: Click the Import Execution Results button.
Step 3: Click the Browse button to navigate the local disk and select the JSON/XML/ZIP file with the automated execution results for the Test Execution.
Step 4: Click Import.
