A Test Run (sometimes simply referred to as a "run") occurs when a Test is scheduled to execute within a Test Execution. Therefore, each time you run a Test, you're, in fact, performing a Test Run.

A Test Run is an internal instance of a Test scenario that is used to validate a specific version of source code/SUT, in a specific environment. Various test runs allow you to easily detect code defects  that could lead to system failures.

A Test Run is not a Jira issue; it's an Xray internal entity.

Normally, you may see Test Runs within specific panels, such as in a Test Execution (a test run per Test) or within the Test issue screen (history of previous runs for that Test).

What is stored in a Test Run

A Test Run contains a copy of the testing specification of the Test and associated Preconditions. This specification only comprises some fields of these entities; thus, not all fields are copied to the Test Run.

General custom fields, summary, description fields are not replicated in the Test Run. Therefore, avoid using these fields to add "relevant" information to the test specification. 

Depending on the original type of  the Test associated with this Test Run, these fields are replicated in the Test Run:

Since a Test Run is related to the execution of a Test, obviously, it contains comments and information about linked defects and evidences (i.e., attachments).

Data consistency

Test Runs represent the results of running some Test specification at some moment in time, in the context of some Test Execution.  

What happens if you change the Test specification at any time? Your existing Test Runs will not change (unless you want them to). This ensures data consistency and compliance with conformance regulations.

At the execution screen of a given Test Run, you will be presented with two options:

1. Merge current Test specification with the current specification in the Test Run
2. Reset the current Test Run, by copying the current Test specification to the Test Run and discard all recorded results

Execution History

The execution history of a given Test issue is available in the Test issue screen in the "Test Runs" section.

The Test Runs section is headlined by the Test Runs Filter that allows you to filter for:

• the Project in which the Test Execution the current Test is associated with
• the target release Version tested by the Test Execution the current Test is associated with
• the Status of the Test Execution the current Test is associated with
• the Start and End dates of the Test Execution the current Test is associated with
• the Test Environment of the Test Executions the current Test is associated with
• the Clear button that empties all the Test Runs Filter fields

The search results are displayed on the table immediately under the Test Runs Filter, containing the following columns:

• the Execution Key of the listed Test Execution the current Test is associated with
• the Fix Version tested by the listed Test Execution the current Test is associated with
• the System Revision tested by the listed Test Execution the current Test is associated with
• the Exec by who is the Assignee of the listed Test Execution the current Test is associated with
• the Start and End dates of the listed Test Execution the current Test is associated with
• the Defects related with the listed Test Execution the current Test is associated with
• the Status of the listed Test Execution the current Test is associated with

Actions

The Execute Test and Execute Test Inline actions are available in the Test issue screen. They may also be available in similar Test Runs sections that may be present in other places.

Execute Test

To execute a Test from a Test issue:

Step 1: Open the Test you want to run.

Step 2: Click Test Runs tab.

Step 3: Click Run (1, 2) to open the Execution page.

The icon in 2 is only displayed when the test run is in a final status.

Execute Test Inline

To execute Test Runs Inline from the Test issue screen, this option must be enabled in the Xray administration page. The option for setting the Test Run status manually, without having to execute all Test Steps (or Examples in the case of Cucumber Tests), must also be enabled in the Miscellaneous Settings either from the Project Settings or Global Settings.

If the above options are enabled and you have permission to execute the Test, the context menu for executing Test Runs, accessed be the Execute button on each Test Run must show the available transitions. 

When executing inline Tests, the status of the manual steps (or Examples in the case of Cucumber Tests) can be changed automatically. The following rules are applied:

• PASS: All steps statuses will be changed to PASS independent of their current status.
• FAIL: All steps statuses will remain unchanged, even if they are all PASS. This is because a Test might fail due to a Pre-Condition.
• TO DO: All steps statuses will be changed to TO DO.
• EXECUTING: The Test execution status will be changed to EXECUTING.
• CUSTOM: The Test execution status will be changed to the specified custom status.
  

To execute a Test Run inline:

Step 1: Open the Test issue you want to run.

Step 2: Click Test Runs tab.

Step 3: Click on the Status located in the status column of the Test Runs table and select one of the available statuses.

View Test Execution Details

To view the execution details:

Step 1: Open the Test issue you want to view the test execution details.

Step 2: Click Test Runs tab.

Step 3: Click the Execute button, which appears in the last column of the desired Test Run that is in a final state, to open the Execution page and view the details.

Ad hoc Test Execution

To execute a test in an ad hoc manner:

Step 1: Open the Test issue you want to execute.

Step 2: Click Test Runs tab.

Step 3: Click the Execute In button in Test Runs section and select New Test Execution....

Step 4: The dialog comes with pre-populated fields so if you don't want to change anything, just click Create.

Note: If there are custom required fields, an error message will be displayed asking you to open the normal create issue screen with the pre-populated fields. If the revision field is not configured in the create issue screen of the Test Execution for the selected Project, the field will not appear in the dialog.

If the Execute Immediately option is selected, you will be redirected to the Execute Test page after the test execution is created.

Configure Test Run Columns

Xray provides you to configure columns for the Test Runs table. This configuration is specific to each user and can be restored to the default configuration defined in the Default Column Layouts page in Xray's Project Settings.

Some columns may be fixed so you won't be able to remove them from the table.

To configure columns for the Test Runs table,

Step 1: Go to the Test Runs table.

Step 2: Click on the Columns, select the dropdown that appears above the table in the top-right corner. A list of the current configure columns and all available columns appears.

Step 3: Add or remove columns by clicking the checkboxes next to each field.

Step 4: Click Done when finished or Cancel to abort this operation. The table refreshes with the configured columns.

Test Run-Specific Columns

The following columns are not Jira fields, but they represent information stored internally by Xray for Test Runs. You can also choose these columns for your layout.

• TestRun Assignee - The Test Run assignee
• TestRun Comment - The Test Run comment
• TestRun Executed By - The last user to execute the Test Run
• TestRun Started - The Test Run start date
• TestRun Finished - The Test Run finish date
• TestRun Progress - The Test Run progress

Restoring Columns to Default Configuration

To restore columns in the Test Runs table to the default configuration,

Step 1: Open the Test Execution issue.

Step 2: Click Columns and select the dropdown that appears above the table in the top-right corner. A list of the current configure columns and all available columns appears.

Step 3: Click the Restore Defaults link above the columns list. The table refreshes with the default configured columns.

Change Columns Order

To reorder columns in the Test Runs table,

Step 1: Open the Test Execution issue.

Step 2: Hover over the desired column header.

Step 3: Click on the column header and drag it to the desired position.

Archive Test Runs

Xray provides you with a way to explicitly archive test runs one at a time or in bulk as well as way to check all the archived test runs for a specific test. For more information about Test Run Archive check here.

Archive a Single Test Run

To archive a single Test Run,

Step 1: Click on the ... at the end of the row that contains the test run you wish to archive.

Step 2: Click on the Archive option that appears.

Step 3: Click the Archive button on the confirmation dialog.

Bulk archive Test Runs

To archive multiple test runs at the same time,

Step 1: Selected the test runs you wish to archive by clicking on the checkbox at the left of each test run.

Step 2: Click on the bottom that appears at the top of the table and then Archive.

Step 3: Click the Archive button on the confirmation dialog.

View archived Test Runs

To view the archived test runs of a test,

Step 1: Click on the ... next to the Execute In button.

Step 2: Click on the Show Archived... option that appears. After this a dialog will appear that shows all archived test runs for the test you are on.

There are 2 actions that can be performed on the archived Test Runs:

1. Restore - By clicking on the ... at the end of the archived test run row and then Restore.
2. Delete - By clicking on the ... at the end of the archived test run row and then Delete.

Both of these actions can be performed in bulk by selecting the test runs using the checkbox on the left then 

Step 1: Selecting the archived test runs by clicking on the checkbox at the left of each test run.

Step 2: Click on the bottom that appears at the top of the table and then the desired action.

Step 3: Confirm the action.