Introduction
A Test Run (sometimes simply referred to as a "run") occurs when a Test is scheduled for execution within a Test Execution. Therefore, each time you execute a Test, you are performing a Test Run.
A Test Run is an internal instance of a Test scenario used to validate a specific version of source code or SUT (System Under Test) in a particular environment. A Test Run is not a Jira Issue; it's an internal Xray entity.
Multiple Test Runs allow you to detect code defects that could potentially lead to system failures more effectively.
Typically, you may encounter Test Runs within specific panels, such as in a Test Execution (one Test Run per Test) or on the Test Issue screen (which displays the history of previous runs for that Test).
A Test Execution represents a task for executing multiple Tests. It contains as many Test Runs as there are Tests associated with it. Specifically, there is one Test Run per associated Test, allowing you to track the result of each Test individually.
Test Run Advantages
A Test Run is highly beneficial because it allows for the precise tracking and management of individual Test Executions within a specific context. Here’s why a Test Run is advantageous:
Traceability and accountability: a Test Run provides a clear record of when and how a test was executed, including the environment, data used, and the person responsible for the execution. This traceability is crucial for understanding the Testing process and identifying potential Issues or areas for improvement.
Version control: since a Test Run captures the exact definition of a Test at the time of execution, it ensures that the results are tied to the specific version of the Test that was run. This is important because it prevents confusion if the Test definition changes later; you can always refer back to the exact conditions under which the Test was executed.
Contextual results: each Test Run is associated with a specific Test Execution, meaning that results are contextualized within the broader scope of the testing effort, such as a release or sprint. This allows teams to see how a Test performed under particular conditions or environments, providing more meaningful insights into the product’s quality.
Customizable statuses: Xray allows for the customization of Test Run statuses, enabling teams to define and track the progress of their Tests in a way that aligns with their specific workflow. This flexibility ensures that the status of each Test is accurately reflected, making it easier to manage the overall Testing process.
Granular Reporting: Test Runs facilitate detailed reporting on individual Test outcomes, making it possible to drill down into specific Issues, trends, or successes within a Test Execution. This granularity is vital for stakeholders who need to understand the quality of the product at a detailed level.
Integration with CI/CD: In automated Testing, Test Runs are automatically generated and updated with results from Continuous Integration/Continuous Deployment (CI/CD) pipelines. This seamless integration ensures that the latest test results are always available, helping teams to quickly identify and address Issues in the development process.
Test Run Contents
A Test Run contains a copy of the Test specification and associated Preconditions. However, this specification includes only certain fields from these entities; not all fields are copied to the Test Run.
General custom fields, as well as the summary and description fields, are not replicated in the Test Run. Therefore, avoid using these fields to add important information to the Test specification.
Depending on the original type of the Test associated with this Test Run, the following fields are replicated in the Test Run:
| Manual Tests | Cucumber Tests | Generic Tests | |
|---|---|---|---|
| Test |
|
|
|
| Linked Precondition(s) |
|
|
|
Since a Test Run is related to the execution of a Test it contains comments and information about linked defects and evidence (i.e., attachments).
Data Consistency
Test Runs represent the results of executing specific Test specifications at a particular moment in time, within the context of a Test Execution.
If you change the Test specification at any point, your existing Test Runs will remain unchanged (unless you choose to update them). This approach ensures data consistency and compliance with conformance regulations.
On the execution screen of a given Test Run, you will be presented with two options (Figure 1):
Figure 1 - Reset/merge
- Merging the current Test specification with the current specification in the Test Run.
- Resetting the current Test Run, by copying the current Test specification to the Test Run and discarding all recorded results.
Execution History
The execution history of a given Test Issue is available in the Test Issue screen in the Test Runs History section (Figure 2 - 2).
Figure 2 - Test run
You can filter Test Runs using the Filters options (Figure 2 - 3) and refine your search by:
Figure 3 - Filter
- The Project with which the Test Execution of the current Test is associated with (Figure 3 - 1).
- The target release Version tested by the Test Execution the current Test is associated with (Figure 3 - 2).
- The Status of the Test Execution the current Test is associated with (Figure 3 - 3).
- The Start and End dates of the Test Execution the current Test is associated with (Figure 3 - 4).
- The Test Environment of the Test Executions the current Test is associated with (Figure 3 - 5).
- The Clear button, which empties all the Test Runs Filter fields (Figure 3 - 7).
- You can click More to access to other filtering options (Figure 3 - 6).
- Once you're finished, click Apply (Figure 3 - 8).
The search results are displayed on the table immediately below the Test Runs Filter, containing the following columns:
- The Execution Key of the listed Test Execution the current Test is associated with.
- The Fix Version tested by the listed Test Execution the current Test is associated with.
- The System Revision tested by the listed Test Execution the current Test is associated with.
- The Exec by, who is the Assignee of the listed Test Execution the current Test is associated with.
- The Start and End dates of the listed Test Execution the current Test is associated with.
- The Defects related to the listed Test Execution the current Test is associated with
- the Status of the listed Test Execution the current Test is associated with.
Operations
The Execute Test (Figure 4 - 1) and Execute Test Inline (Figure 4 - 2) actions are available in the Test Issue screen (Figure 4).
Figure 4 - Executing
Executing a Test
From a Test Issue
Open the Test you want to run.
Click Test Runs (Figure 4 - 6).
Click the Run icon (Figure 4 - 2; 3) to open the Execution screen (Figure 5). The icon 3 (Figure 4 - 3) is only displayed when the Test run is in a final status.
Inline
To execute Test Runs inline from the Test Issue screen (Figure 4), this option must first be enabled in the Xray administration screen. Additionally, the option to set the Test Run status manually — without executing all Test Steps (or Examples in the case of Cucumber Tests)—must be enabled in the Miscellaneous Settings, either in the Project Settings or Global Settings.
If these options are enabled and you have permission to execute the Test, the context menu for executing Test Runs, accessed via the Execute button on each Test Run, will display the available transitions.
When executing Tests inline, the status of manual Steps (or Examples in the case of Cucumber Tests) can be changed automatically according to the following rules:
- PASSED: all Steps statuses will be changed to PASSED, regardless of their current status.
- FAILED: all Steps statuses will remain unchanged, even if they are all PASSED, because a Test might fail due to a Precondition.
- TO DO: all Steps statuses will be changed to TO DO.
- EXECUTING: the Test execution status will be changed to EXECUTING.
- CUSTOM: The Test execution status will be changed to the specified custom status.
Open the Test you want to run (Figure 4).
Click Test Runs (Figure 4 - 6).
Click a Status to see the other options and change it if you want to (Figure 4 - 4).
Click the play/run icon (Figure 4 - 2) to open the Execution screen and start the execution (Figure 5)
Figure 5 - Execution screen
The icon 3 (Figure 4 - 3) is only displayed when the Test Run is in a final status.
Ad Hoc Test Execution
Open the Test you want to run (Figure 6).
Click Test Runs (Figure 6 - 1).
Figure 6 - Executing
Click Execute in (Figure 6 - 2). A dropdown menu will open. Select the New test execution option (Figure 6 - 3).
A modal will open (Figure 7).
Figure 7 - Creating
The modal comes with pre-populated fields (Figure 7 - 1), so if you don't want to change anything, just click Create (Figure 7 - 3).
If there are custom required fields, an error message will be displayed asking you to open the regular Create Issue screen with the pre-populated fields. If the revision field is not configured in the Create Issue screen of the Test Execution for the selected Project, the field will not appear in the modal.
If the Execute Immediately option is selected (Figure 7 - 2), you will be redirected to the Execute Test screen (Figure 6) after the Test execution is created.
Viewing the Test Execution Details
Open the Test you want to run (Figure 8).
Figure 8 - Test
Click Test Runs (Figure 8 - 1).
Click the details icon (Figure 8 - 3) to open the Execution screen and see all its details (Figure 9).
Figure 9 - Execution
The icon 3 (Figure 8 - 3) is only displayed when the Test Run is in a final status.
Configuring Test Run Columns
Xray allows you to configure columns for the Test Runs table (Figure 10). This configuration is specific to each user and can be restored to the default configuration defined in the Default Column Layouts screen, in Xray's Project Settings.
Configuring Test Run columns in Xray is useful because it allows you to customize the view to display only the most relevant information for your specific testing needs. By selecting and organizing columns that show key details such as status, assignee, or environment, you can streamline the management of Test Runs, improve focus on critical data, and enhance overall efficiency in tracking Test progress and outcomes.
Some columns may be fixed so you won't be able to remove them from the table.
To configure columns for the Test Runs table:
Go to the Test Runs table (Figure 10).
Figure 10 - Columns
Click Columns (Figure 10 - 1).
A dropdown menu will open with a list of the currently configured columns, and all available columns will appear. You can find more by using the search bar (Figure 10 - 2).
Select your option/s by checking the corresponding box (Figure 10 - 4).
Once you're finished, click Done (Figure 10 - 5). The table refreshes with the newly configured columns.
You can restore the default columns by clicking Restore Defaults (Figure 10 - 3).
Test Run-Specific Columns
The following columns (Figure 11 - 2) are not Jira fields, but they represent information stored internally by Xray for Test Runs.
Figure 11 - Columns
You can also choose the columns below for your layout:
To access them, click Columns (Figure 11 - 1) and scroll down (Figure 11 - 2). Once you're finished selecting the columns, click Done (Figure 11 - 3).
Changing the Columns Order
Go to the Test Runs table (Figure 12).
Figure 12 - Columns
Hover over the desired column header (Figure 12 - 1).
Click the column header (Figure 12 - 1) and drag it to the desired position.
Restoring Columns to Default Configuration
Open the Test Execution Issue (Figure 13).
Figure 13 - Restore
Click Columns (Figure 13 - 1). A dropdown menu will open. There, click Restore Defaults (Figure 13 - 2). The table refreshes with the default configured columns.
Archiving Test Runs
Xray allows you to archive Test Runs one at a time or in bulk, as well as a way to check all the archived Test runs for a specific Test.
Archiving Test Runs is useful because it helps you maintain a clean and organized workspace by removing outdated or irrelevant Test data from active view. This process enhances performance by reducing clutter in your Test management environment, and allows you to better focus on current Test Runs. Additionally, archived Test Runs are preserved for historical reference and audit purposes, ensuring that past Test data can still be accessed if needed, without interfering with ongoing testing activities.
Archiving a Single Test Run
Go to the Test Runs table (Figure 14).
Figure 14 - Archive
On the desired row, click the Actions icon (Figure 14 - 1) and then select Archive (Figure 14 - 2).
A modal (Figure 15) will open prompting you to confirm the archiving of the Test Run. Click the Archive button (Figure 15 - 1) to proceed.
Figure 15 - Modal
Bulk-Archiving Test Runs
Go to the Test Runs table (Figure 16).
Check the boxes corresponding to the rows of the Tests you wish to archive (Figure 16 - 1).
Figure 16 - Bulk archive
Once you've checked the boxes, the Selected button (Figure 16 - 2) will appear. Click it and you will see a dropdown menu. There, click Archive (Figure 16 - 3).
A modal will open for you to confirm this operation.
Click Archive (Figure 17 - 1) to proceed.
Figure 17 - Archive
Viewing Archived Test Runs
On the Test Issue, click the ellipsis button (Figure 18 - 1) and then click Show archived... (Figure 18 - 2).
Figure 18 - Archived
A modal will open, showing all archived Test Runs for the Test you are on (Figure 19).
In this modal (Figure 19) you can click the ellipsis button (Figure 19 - 1) and then restore/delete (Figure 19 - 2; 3) that Test Run.
Figure 19 - Archived
Both of these actions can be performed in bulk by checking the boxes (Figure 19 - 5) and then clicking the restore/delete options (Figure 19 - 3).
Click Close (Figure 19 - 4) to go back to the Test Issue (Figure 18).
You can also execute, assign, and remove Test Runs (Figure 16 - 3).