A Test Run (sometimes simply referred to as a "run") occurs when a Test is scheduled to execute within a Test Execution. Therefore, each time you run a Test, you're, in fact, running a Test Run.

A Test Run is an internal instance of a Test scenario that is used to validate some specific version of source code/SUT, in a specific environment. Various test runs allow you to easily detect code defects  that could lead to system failures.

A Test Run is not a Jira issue; it's an Xray internal entity that you can search (see Enhanced querying with JQL).

Normally, you may see Test Runs within specific panels, such as in a Test Execution (a test run per Test) or within the Test issue screen (history of previous runs for that Test).

What is stored in a Test Run

A Test Run contains a copy of the testing specification of the Test and associated Pre-Conditions. This specification only comprises some fields of these entities; thus, not all fields are copied to the Test Run.

General custom fields, summary, description fields are not replicated in the Test Run. Thus, you should avoid using those fields to add "relevant" information to the test specification. 

Depending on the original type of  the Test associated with this Test Run, these are the fields are replicated in the Test Run.

Since a Test Run is related to the execution of a Test, obviously, it contains comments and information about linked defects and evidences (i.e., attachments).

Data consistency

Test Runs represent the results of running some Test specification at some moment in time, in the context of some Test Execution. Thus, they assure data consistency. 

So, what happens if you change the Test specification at any time? Your existing Test Runs will not change (unless you want them to), in order to ensure data consistency and compliance with conformance regulations.

At the execution screen of a given Test Run, you will be presented with two options:

1. Merge current Test specification with the current specification in the Test Run
2. Reset the current Test Run, by copying the current Test specification to the Test Run and discard all recorded results

Execution History

The execution history of a given Test issue is available in the Test issue screen in the "Test Runs" section.

The Test Runs section is headlined by the Test Runs Filter that allows you to sort for:

• the Project in which the Test Execution the current Test is associated with
• the target release Version tested by the Test Execution the current Test is associated with
• the Status of the Test Execution the current Test is associated with
• the Start and End dates of the Test Execution the current Test is associated with
• the Clear button, that empties all the Test Runs Filter fields

The search results are displayed on the table immediately under the Test Runs Filter, containing the following columns:

• the Execution Key of the listed Test Execution the current Test is associated with
• the Fix Version tested by the listed Test Execution the current Test is associated with
• the System Revision tested by the listed Test Execution the current Test is associated with
• the Exec by who is the Assignee of the listed Test Execution the current Test is associated with
• the Start and End dates of the listed Test Execution the current Test is associated with
• the Defects related with the listed Test Execution the current Test is associated with
• the Status of the listed Test Execution the current Test is associated with

Actions

The "Execute Test" and "Execute Test Inline" actions are available in the Test issue screen. They may also be available in similar Test Runs sections that may be present in other places.

Execute Test

To execute a Test from a Test issue:

Step 1: Open the Test you wish to run.

Step 2: Click the Execute button that appears in the last column of the desired Test Run. The actions menu should pop up with the available actions.

Step 3: Click Run action to open the Execution page.

Execute Test Inline

To execute Test Runs Inline from the Test issue screen, this option must be enabled in the Xray administration page. The option for setting the Test Run status manually, without having to execute all Test Steps (or Examples in the case of Cucumber Tests), must also be enabled in the Xray administration page.

If the above options are enabled and you have permission to execute the Test, the context menu for executing Test Runs, accessed be the Execute button on each Test Run must show the available transitions. 

When executing inline Tests, the status of the manual steps (or Examples in the case of Cucumber Tests), can be changed automatically. The following rules are applied:

• PASS: All steps statuses will be changed to PASS independent of their current status.
• FAIL: All steps statuses will remain unchanged, even if they are all PASS. This is because a Test might fail due to a Pre-Condition.
• TODO: All steps statuses will be changed to TODO.  
• ABORTED: The Test execution status will be changed to ABORTED. 
• EXECUTING: The Test execution status will be changed to EXECUTING.
• CUSTOM: The Test execution status will be changed to the specified custom status.
  

To execute a Test Run inline:

Step 1: Open the Test issue you wish to run.

Step 2: Click the Execute button located in the last column of the Test Runs table and select one of the available statuses.

 

View Test Execution Details

To view the execution details:

Step 1: Open the Test issue you wish to view the test execution details.

Step 2: Click the Execute button that appears in the last column of the desired Test Run that is in a final state. The actions menu should pop up with the available actions.

Step 3: Click Execution Details action to open the Execution page and view the details.

Ad hoc Test Execution

To execute a test in an ad hoc manner:

Step 1: Open the Test issue you wish to execute.

Step 2: Click the Execute In button in Test Runs section and select New Test Execution....

Step 3: The Jira create issue dialog to create Test Execution comes with pre-populated fields so if you don't want to change anything, just click Create.

Note: If the revision field is not configured in the create issue screen of the Test Execution for the selected Project, the field will not appear in the dialog.

If the Redirect To Execute Test Page option is selected, you will be redirected to the Execute Test page after the test execution is created.

Configure Test Run Columns

Xray provides the ability to configure columns for the Test Runs table. This configuration is specific to each user and can be restored to the default configuration defined in the Global Preferences page in Xray's administration.

Some columns may be be fixed and thus you won't be able to remove them from the table.

To configure columns for the Test Runs table,

Step 1: Go to the Test Runs table.

Step 2: Click on the Columns, select the dropdown that appears above the table in the top-right corner. A list of the current configure columns and all available columns appears.

Step 3: Add or remove columns by clicking the checkboxes next to each field.

Step 4: Click the Done button when finished or Cancel to abort this operation. The table refreshes with the configured columns.

Test Run-Specific Columns

The following columns are not Jira fields, but they represent information stored internally by Xray for Test Runs. You can also choose these columns for your layout.

• TestRun Assignee - The Test Run assignee
• TestRun Started - The Test Run start date
• TestRun Finished - The Test Run finish date
• TestRun Executed By - The last user to execute the Test Run
• Test Run Progress - The manual steps progress, formatted as  "%percent (finished_steps/total_steps)", e.g. "50% (5/10)". Only steps whose status contributes to a final Test Run status will be considered (i.e. completed steps). In case the Test has a dataset, then this field will be formatted as "%percent (finished_iterations/total_iterations)"

Test Run Custom Fields

Test Run Custom Fields can be added to the table. If the test execution project has any Test Run Custom Field then a user can add them to the Test Run table in a Test Execution Page.

Restoring Columns to Default Configuration

To restore columns in the Test Runs table to the default configuration,

Step 1: Open the Test Execution issue.

Step 2: Click on the Columns, select the dropdown that appears above the table in the top-right corner. A list of the current configure columns and all available columns appear.

Step 3: Click the Restore Columns link above the columns list. The table refreshes with the default configured columns.

Change Columns Order

To reorder columns in the Test Runs table,

Step 1: Open the Test Execution issue.

Step 2: Hover over the desired column header.

Step 3: Click on the column header and drag it to the desired position.