Overview

You can either execute your tests manually or automatically. If it is an automated test, you can still manually set the test status for that execution.You can also create custom statuses. 

...

A Test Execution validates if all the Tests in a specific environment target passed. In order to execute, monitor the progress, and report the results, each Test has its own Execution page details. Each Test associated with a Test Execution is called a Test Run. A Test Run is like an instance of the Test that contains a copy of the test definition (e.g., description, steps) and the result of that test in the context of that Test Execution.

...

• you have permission to "Browse" the Test Execution issue
• you have permission to "Resolve" the Test Execution issue
• the Test Execution issue is not in a status configured to disallow executions
• the Test issue is not in a status configured to disallow executions

Even though you might not have permission to execute a Test Run and if the execution has already started, you can see the execution details page in read-only mode. An informative message will be shown to at the top of the execution page if the current Test Run is in read-only mode.

...

When you execute a Test, the Xray Execution Screen is shown. This screen is accessible from the Test Execution issue view page and the Test issue view page (Test Runs section), and this is where the execution results are displayed and registered. This page contains information about the Test definition and the Test Execution issue context to allow you to execute the Test without exiting this page.

The execution screen is composed by of the following sections:toc

• Navigation and Actions bar
• Execution Status and Test Execution fields
• Comments
• Defects
• Evidence
• Test issue details
• Custom Fields
• Preconditions
• Test details (script or steps)
• Result contexts
• Activity

Navigation and Actions bar

This bar is always available to allow you to navigate the Test issue hierarchy. It also contains a toolbar with the Return to Test Execution button that allows you to jump to the current Test Execution issue page. The Next and Previous buttons allow you to quickly navigate to the execution screen of the other Tests in the same Test Execution, if applicable.

Image Removed

Export Automated Test Definition

If you are executing an automated Test, you can export its definition right from the execution screen.

On this bar, you can also find actions to Export test definitions and Import execution results. This is useful if you need to do an ad-hoc execution for an automated Test. Usually, automated tests are executed in a Continuos Integrating platform and the results are imported automatically into Xray (creating new Test Execution issues for the results) but you can also execute each automated Test individually and import the execution result.

Image Added

Export to Cucumber

If you're executing a Cucumber Test, an option to export the Test as a feature file will appear.Image Removed

Import Execution Results

If you already have the test results, just use this action to import the report files into Xray and update a specific Test Run.

Execution Status

The Execution Status describes the current Test progress. The possible (native) status for a Test Run are TODO, EXECUTING, FAIL, and PASS. You can also create custom statuses in the Xray Test Statuses Settings global page.

...

If the dates are configured to be edited, you can edit the Started On date by clicking the calendar icon and choosing the date and time. You can check your configurations by going to Jira Administration > Add-ons > Miscellaneous.

Comment

This field is inline editable so you can input comments about the current execution without reloading the page.

It supports the Jira markup wiki language.

Image Added

Execution Defects

This field lists the overall defects related to the current execution. It is possible to Create new Defect issues (or sub-tasks) or Add existing Defects by clicking the respective option.

Execution

...

Evidence

This field lists the attachments related with to the current execution. Clicking Add Evidences opens up the Attach Files dialog box. Check here for more information about attachments.

Comment

This field is inline editable so you can input comments about the current execution without reloading the page.

It supports the Jira markup wiki language.

Image Removed

Issue Details

This section includes some fields present on the Test issue. It often contains useful information for testers to execute the Test.

Test Description

This section includes the current Test issue description. It often contains useful information for testers to execute the Test.

...

This section shows all attachments associated with the Test. 

...

Custom Fields

You define additional custom fields for Test Runs. These fields can be useful to add extra information to Test Runs, usually only available when or after executing Tests.

Test Run custom fields can be configured by project and by Test Type. Therefore, these settings will not affect other projects within your Jira instance. For instance, it is possible to have custom fields just for Manual Tests within a project.

If there are any custom fields defined for the scope of the Test Run, they will appear on the "Custom Fields" section. 

Image Added

It is also possible to define custom fields that are mandatory. If this is the case, users will not be able to set a final status on the Test Run if there is any required custom field that is not populated.

Image Added

Preconditions

This field shows the Preconditions issues associated with the current Test.

Test Details

Within the Test Details section, you will find the Test script to execute. Depending on the Test Type, this section can display Test Steps, a Generic Definition field (for unstructured Tests), and Gherkin scenarios.

Manual Test Step Results Table

A Manual Test execution screen contains the Test Steps, with the Action, attachments and all the other configured fields to be used as a reference while testing. The Test Steps field configuration used is the one defined in the Test Run's Test project.

...

• the Actual Result field which contains the actual result of the execution of the step;
  • This field can be edited using the WYSIWYG markdown editor.
  • It's possible to refer to an Evidence previously added to any Step or Global execution
• the Comment field, which contains any notes about the corresponding step;
• the Defects field, which contains any defect associated with the corresponding step. It is also possible to input new defects by clicking on:
  • Create Defect to create a new Defect
  • Create Sub-Defect to create a new Sub-Defect (sub-task)
  • Add Defects to add an already existing defect
• the Evidences field, which contains any attachment associated with the corresponding step. It is also possible to input new attachments by clicking on the + button, which opens the Attach Files dialogue box
• the Status, which contains the current step testing status. This can be changed by clicking on any of the status statuses in front of your Step Status.

Note: Changing the Status on the each Step will affect the Execution Status field on the Execution Screen.

...

Gherkin Test Details

An Automated Test execution screen contains the Automated Test Details field, which list the Steps to be used as reference by the external tool while testingGherkin Tests always display the Test scenario on the execution page. Usually, these tests are automated and their result can be imported into Xray. However, it is also possible to execute these Tests manually by setting the overall result of the Test Run, or by setting the result of each example (in case of scenario outlines).

Examples

A scenario outline of an automated Cucumber test will normally contain an examples table containing scenario outline contains a table of examples that define several inputs and outputs for the given scenario. Each example is, therefore, a set of values that will be replaced on the scenario.

The examples section details these examples, containing an overall status bar in the header and a table with all the examples below. Each example can have the status: PASS, FAIL, EXECUTING or TODO. 

...

A scenario outline test is TODO if at least one of the examples did not run and all the other examples passed or todo.

Unstructured Test Details

Unstructured Tests only contain the definition field on the Test Details section. This field can specify a test script, an ID to an external (automated) test case, or an Exploratory Test charter. Similar to Gherkin Tests, most unstructured test results will be imported or set by another tool. However, it is also possible to set the status of unstructured tests manually.

Image Added

Automated Results

Xray supports the import of multiple execution results for the same Test Run. Multiple execution executions for the same Test Run specify specifies different results for different contextcontexts/environments. Multiple executions are only supported for Automated Tests. Manual Tests do not have this concept, yet.

When a Test Run contains multiple executions imported from external sources like Calabash and Xamarin Test Cloud, these executions are presented in a "Results" section on the execution page.

...

• Context: An execution context name (in the case of Xamarin Test Cloud results, the mobile device, and operating system version) 
• Error Message: A possible error message if the result has failed
• Duration: The complete duration of an execution result
• Status: The execution results status

Activity

The Activity field lists all the activities related to the current execution.

Image Added

Attaching Screenshots

To attach a screenshot, paste the image from your clipboard using the relevant keyboard shortcut. (For Mac OSX, use CMD+V; for Windows, use Ctrl+V.)

...

Click Save to attach the captured image to your Test Run.

Activity

The Activity field lists all the activities related with the current execution.Image Removed

Updating the definition of the Test after the execution started

When a Test definition has changed, a new dialog box will appear at the top of the Execution page informing you that the current execution is obsolete. You will be prompted with two buttons: Reset and Merge. 

Reset Execution

You can choose to reset the Execution or keep it for historic record. If you reset the execution, all information previously saved (status, evidencesevidence, defects) will be erased. 

Merge Execution

You can choose to merge the new Test definition into the current status of execution. This will keep steps that weren't changed along with the associated results/comments/evidencesevidence. It will also load new steps (moved, erased, or created), preconditions, or attachments definitions. This will always force a recalculation of Test Run Status. For instance, if the current Test Run Status is FAIL because of a step and this step is removed from Test, if you merge, then the Test Run Status will change to whatever the value combination of the others steps returns.

...