Versions Compared

Old Version 11

changes.mady.by.user Ivan Filippov

Saved on

compared with

New Version Current

changes.mady.by.user Ana Duarte

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
UI Expand
titleTable of Contents

Table of Contents

Introduction

A Test Run (sometimes simply referred to as a "run") occurs when a Test is scheduled to execute for execution within a Test Execution. Therefore, each time you run execute a Test, you 're, in fact, are performing a Test Run.

A Test Run is an internal instance of a Test scenario that is used to validate a specific version of source code /SUT, or SUT (System Under Test) in a specific environment. Various test runs allow you to easily detect code defects  that could lead to system failures.particular environment. A Test Run is not a Jira issueIssue; it's an internal Xray internal entity.

Multiple Test Runs allow you to detect code defects that could potentially lead to system failures more effectively.

TypicallyNormally, you may see encounter Test Runs within specific panels, such as in a Test Execution (a test run one Test Run per Test) or within on the Test issue Issue screen (which displays the history of previous runs for that Test).

Info

A Test Execution represents a task for running several executing multiple Tests. It contains as many as many Test Runs as the number of Tests associated with there are Tests associated with it. To be preciseSpecifically, there is one Test Run per associated Test, in order allowing you to track the result of each Test individually.

Test Run Advantages

A Test Run is highly beneficial because it allows for the precise tracking and management of individual Test Executions within a specific context. Here’s why a Test Run is advantageous:

  • Traceability and accountability: a Test Run provides a clear record of when and how a test was executed, including the environment, data used, and the person responsible for the execution. This traceability is crucial for understanding the Testing process and identifying potential Issues or areas for improvement.

  • Version control: since a Test Run captures the exact definition of a Test at the time of execution, it ensures that the results are tied to the specific version of the Test that was run. This is important because it prevents confusion if the Test definition changes later; you can always refer back to the exact conditions under which the Test was executed.

  • Contextual results: each Test

...

  • Run is associated with a specific Test Execution, meaning that results are contextualized within the broader scope of the testing effort, such as a release or sprint. This allows teams to see how a Test performed under particular conditions or environments, providing more meaningful insights into the product’s quality.

  • Customizable statuses: Xray allows for the customization of Test Run statuses, enabling teams to define and track the progress of their Tests in a way that aligns with their specific workflow. This flexibility ensures that the status of each Test is accurately reflected, making it easier to manage the overall Testing process.

  • Granular Reporting: Test Runs facilitate detailed reporting on individual Test outcomes, making it possible to drill down into specific Issues, trends, or successes within a Test Execution. This granularity is vital for stakeholders who need to understand the quality of the product at a detailed level.

  • Integration with CI/CD: In automated Testing, Test Runs are automatically generated and updated with results from Continuous Integration/Continuous Deployment (CI/CD) pipelines. This seamless integration ensures that the latest test results are always available, helping teams to quickly identify and address Issues in the development process.

Test Run Contents

Table of Contents

...

A Test Run contains a copy of the testing Test specification of the Test and associated associated Preconditions.  This However, this specification includes only comprises some certain fields of from these entities; thus, not all fields are copied to the Test Run.

General custom fields, as well as the summary , and description fields, are not replicated in the Test Run. Therefore, avoid using these fields to add "relevant" important information to the test Test specification. 

Depending on the original type of  of the Test associated with this Test Run, these the following fields are replicated in the Test Run:


Manual TestsCucumber TestsGeneric Tests
Test
  • Test Type
  • Manual Steps
  • Test Type
  • Scenario
  • Test Type
  • Generic Test Definition
linked
Linked Precondition(s)
  • Precondition(s) Type
  • Conditions
  • Precondition(s) Type
  • Conditions
  • Precondition(s) Type
  • Conditions

Since a Test Run is related to the execution of a Test , obviously, it contains comments and information about linked defects and evidences evidence (i.e., attachments).

Data

...

Consistency

Test Runs represent the results of running some executing specific Test specification specifications at some a particular moment in time, in within the context of some a Test Execution.  

If you What happens if you change the Test specification at any time? Your point, your existing Test Runs will not change remain unchanged (unless you want choose to update them to). This approach ensures data consistency and compliance with conformance regulations.

At On the execution screen of a given Test Run, you will be presented with two options (Figure 1):

...

Image Added

Figure 1 - Reset/merge

  • Merging the current Test specification with the current specification in the Test Run.

...

  • Resetting the current Test Run, by copying the current Test specification to the Test Run and

...

  • discarding all recorded results

...

titleLearn more

...

  • .

Execution History

The execution history of a given Test issue Issue is available in the Test issue Issue screen in the " Test Runs " section.

The Test Runs section is headlined by the Test Runs Filter that allows you to filter for:

History section (Figure 2 - 2).

Figure 2 - Test runImage Added

Figure 2 - Test run


You can filter Test Runs using the Filters options (Figure 2 - 3) and refine your search by:

Figure 3 - FilterImage Added

Figure 3 - Filter

  • The Project with the Project in which the Test Execution of the current Test is associated with (Figure 3 - 1).
  • The target release Version tested the target release Version tested by the Test Execution the current Test is associated withis associated with (Figure 3 - 2).
  • The Status of the Status of the Test Execution the current Test is associated withis associated with (Figure 3 - 3).
  • The the Start and End dates dates of the Test the Test Execution the current Test is associated withis associated with (Figure 3 - 4).
  • The the Test Environment of the Test Executions the current Test is associated with (Figure 3 - 5).
  • The Clear button, which the Clear button that empties all the Test Runs Filter fields (Figure 3 - 7).
  • You can click More to access to other filtering options (Figure 3 - 6).
  • Once you're finished, click Apply (Figure 3 - 8).

The search results are displayed on the table immediately under immediately below the Test Runs Filter, containing the following columns:

  • the The Execution Key of of the listed Test Execution the current Test is associated with.
  • the The Fix Version tested tested by the listed Test Execution the current Test is associated with.
  • the The System Revision tested tested by the listed Test Execution the current Test is associated with.
  • the The Exec by who , who is the Assignee of the listed Test Execution the current Test is associated with.
  • the The Start and End dates dates of the listed Test Execution the current Test is associated with.
  • the The Defects related with the related to the listed Test Execution the current Test is associated with
  • the Status of of the listed Test Execution the current Test is associated with.

...

Operations

The Execute Test (Figure 4 - 1) and Execute Test Inline (Figure 4 - 2) actions are available in the Test issue Issue screen . They may also be available in similar Test Runs sections that may be present in other places.

Table of Contents
maxLevel2
minLevel2

Execute Test

To execute a Test from a Test issue:

...

(Figure 4). 

Image Added

Figure 4 - Executing

Executing a Test

From a Test Issue

UI Steps
UI Step

Open the Test you want to run.

...

Step 2: Click Test Runs tab.

Step 3: Click Run (1, 2) to open the Execution page.

Image Removed

...

UI Step

Click Test Runs (Figure 4 - 6).

UI Step

Click the Run icon (Figure 4 - 2; 3) to open the Execution screen (Figure 5). The icon 3 (Figure 4 - 3) is only displayed when the

...

Test run is in a final status.

...

Inline

To execute Test Runs Inline inline from the Test issue screenthe Test Issue screen (Figure 4), this option must first be enabled in the Xray administration pagescreen. The option for setting Additionally, the option to set the Test Run status manually , — without having to execute executing all Test Steps (or Examples in the case of Cucumber Tests), must also —must be enabled in the Miscellaneous Settings, either from in the Project Settings or Global Settings.

If the above these options are enabled and you have permission to execute the Test, the context menu for executing Test Runs, accessed be via the Execute  Execute button on each Test Run must show , will display the available transitions. 

When executing Tests inline Tests, the status of the manual steps Steps (or Examples in the case of Cucumber Tests) can be changed automatically . The according to the following rules are applied:

  • PASSPASSED: All steps all Steps statuses will be changed to PASS independent of PASSED, regardless of their current status.
  • FAILFAILED: All steps all Steps statuses will remain unchanged, even if they are all PASS. This is PASSED, because a Test might fail due to a Pre-ConditionPrecondition.
  • TO DO: All steps all Steps statuses will be changed to TO DO.
  • EXECUTING: The the Test execution status will be changed to EXECUTING.
  • CUSTOM: The Test execution status will be changed to the specified custom status.

To execute a Test Run inline:

...

UI Steps
UI Step

Open the Test

...

you want to run (Figure 4).

...

Step 2: Click Test Runs tab.

Step 3: Click on the Status located in the status column of the Test Runs table and select one of the available statuses.

Image Removed

View Test Execution Details

To view the execution details:

Step 1: Open the Test issue you want to view the test execution details.

Step 2: Click Test Runs tab.

Step 3: Click the Execute button, which appears in the last column of the desired Test Run that is in a final state, to open the Execution page and view the details.

Image Removed

Ad hoc Test Execution

To execute a test in an ad hoc manner:

Step 1: Open the Test issue you want to execute.

Step 2: Click Test Runs tab.

Step 3: Click the Execute In button in Test Runs section and select New Test Execution....

Image Removed

...

UI Step

Click Test Runs (Figure 4 - 6).

UI Step

Click a Status to see the other options and change it if you want to (Figure 4 - 4).

UI Step

Click the play/run icon (Figure 4 - 2) to open the Execution screen and start the execution (Figure 5)

Figure 5 - Execution screenImage Added

Figure 5 - Execution screen


Info

The icon 3 (Figure 4 - 3) is only displayed when the Test Run is in a final status.

Ad Hoc Test Execution

UI Steps
UI Step

Open the Test you want to run (Figure 6).

UI Step

Click Test Runs (Figure 6 - 1).

Figure 6 - ExecutingImage Added

Figure 6 - Executing

UI Step

Click Execute in (Figure 6 - 2). A dropdown menu will open. Select the New test execution option (Figure 6 - 3).

UI Step

 A modal will open (Figure 7).

Figure 7 - CreatingImage Added

Figure 7 - Creating


The modal comes with pre-populated fields (Figure 7 - 1), so if you don't want to change anything, just

...

click Create (Figure 7 - 3).

...

If there are custom required fields, an error message will be displayed asking you to open the

...

regular Create Issue screen with the pre-populated fields. If the revision field is not configured in the

...

Create Issue screen of the Test Execution for the selected Project, the field will not appear in the

...

modal.

Image Removed

If the Execute Immediately

...

option is selected (Figure 7 - 2), you will be redirected to the Execute Test

...

screen (Figure 6) after the

...

Test execution is created.

Viewing the Test Execution Details

UI Steps
UI Step

Open the Test you want to run (Figure 8).

Figure 8 - TestImage Added

Figure 8 - Test

UI Step

Click Test Runs (Figure 8 - 1).

UI Step

Click the details icon (Figure 8 - 3) to open the Execution screen and see all its details (Figure 9).

...

Figure 9 - ExecutionImage Added

Figure 9 - Execution

Info

The icon 3 (Figure 8 - 3) is only displayed when the Test Run is in a final status.

Configuring

...

Test Run Columns

Xray provides allows you to configure columns for the Test Runs table (Figure 10). This configuration is specific to specific to each user and can be restored to the default configuration defined in the the Default Column Layouts page screen, in Xray's Project Settings.

Configuring Test Run columns in Xray is useful because it allows you to customize the view to display only the most relevant information for your specific testing needs. By selecting and organizing columns that show key details such as status, assignee, or environment, you can streamline the management of Test Runs, improve focus on critical data, and enhance overall efficiency in tracking Test progress and outcomes.

Info

Some columns may be fixed so you won't be able to remove them from the table.

To configure columns for the Test Runs table,

...

:

UI Steps
UI Step

Go to the Test Runs table (Figure 10).

...

Step 2: Click on the Columns, select the dropdown that appears above the table in the top-right corner. A list of the current configure columns and all available columns appears.

Step 3: Add or remove columns by clicking the checkboxes next to each field.

...

Figure 10 - ColumnsImage Added

Figure 10 - Columns

UI Step

Click Columns (Figure 10 - 1).

A dropdown menu will open with a list of the currently configured columns, and all available columns will appear. You can find more by using the search bar (Figure 10 - 2).

UI Step

Select your option/s by checking the corresponding box (Figure 10 - 4). 

UI Step

Once you're finished, click Done (Figure 10 - 5). The table refreshes with the newly configured columns.

Info

You can restore the default columns by clicking Restore Defaults (Figure 10 - 3).

Test Run-Specific Columns

The following columns (Figure 11 - 2) are not Jira fieldsJira fields, but they but they represent information stored internally by Xray for Test Runs.

Figure 11 - ColumnsImage Added

Figure 11 - Columns


You can also choose these the columns below for your layout.: 

  • The

Restoring Columns to Default Configuration

To restore columns in the Test Runs table to the default configuration,

Step 1: Open the Test Execution issue.

Step 2: Click Columns and select the dropdown that appears above the table in the top-right corner. A list of the current configure columns and all available columns appears.

Step 3: Click the Restore Defaults link above the columns list. The table refreshes with the default configured columns.

Change Columns Order

To reorder columns in the Test Runs table,

Step 1: Open the Test Execution issue.

...

To access them, click Columns (Figure 11 - 1) and scroll down (Figure 11 - 2). Once you're finished selecting the columns, click Done (Figure 11 - 3).

Changing the Columns Order

UI Steps
UI Step

Go to the Test Runs table (Figure 12).

Figure 12 - ColumnsImage Added

Figure 12 - Columns

UI Step

Hover over the desired column header (Figure 12 - 1).

...

UI Step

Click the column header (Figure 12 - 1) and drag it to the desired position.

...

Restoring Columns to Default Configuration

UI Steps
UI Step

Open the Test Execution Issue (Figure 13).

Figure 13 - RestoreImage Added

Figure 13 - Restore

UI Step

Click Columns (Figure 13 - 1). A dropdown menu will open. There, click Restore Defaults (Figure 13 - 2). The table refreshes with the default configured columns.

Archiving Test Runs

Xray provides allows you with a way to explicitly archive test runs Test Runs one at a time or in bulk, as well as a way to check all the archived test Test runs for a specific test. For more information about Test Run Archive check here.

Archive a Single Test Run

To archive a single Test Run,

Step 1: Click on the ... at the end of the row that contains the test run you wish to archive.

Step 2: Click on the Archive option that appears.

Step 3: Click the Archive button on the confirmation dialog.

Image Removed

Image Removed

Bulk archive Test Runs

To archive multiple test runs at the same time,

Step 1: Selected the test runs you wish to archive by clicking on the checkbox at the left of each test run.

Step 2: Click on the bottom that appears at the top of the table and then Archive.

Step 3: Click the Archive button on the confirmation dialog.

Image Removed

Image Removed

View archived Test Runs

To view the archived test runs of a test,

Step 1: Click on the ... next to the Execute In button.

Step 2: Click on the Show Archived... option that appears. After this a dialog will appear that shows all archived test runs for the test you are on.

Image Removed

There are 2 actions that can be performed on the archived Test Runs:

  1. Restore - By clicking on the ... at the end of the archived test run row and then Restore.
  2. Delete - By clicking on the ... at the end of the archived test run row and then Delete.

Test. 

Archiving Test Runs is useful because it helps you maintain a clean and organized workspace by removing outdated or irrelevant Test data from active view. This process enhances performance by reducing clutter in your Test management environment, and allows you to better focus on current Test Runs. Additionally, archived Test Runs are preserved for historical reference and audit purposes, ensuring that past Test data can still be accessed if needed, without interfering with ongoing testing activities.

Archiving a Single Test Run

UI Steps
UI Step

Go to the Test Runs table (Figure 14).

Figure 14 - ArchiveImage Added

Figure 14 - Archive

UI Step

On the desired row, click the Actions icon (Figure 14 - 1) and then select Archive (Figure 14 - 2).

UI Step

A modal (Figure 15) will open prompting you to confirm the archiving of the Test Run. Click the Archive button (Figure 15 - 1) to proceed.

Figure 15 - ModalImage Added

Figure 15 - Modal

Bulk-Archiving Test Runs

UI Steps
UI Step

Go to the Test Runs table (Figure 16).

UI Step

Check the boxes corresponding to the rows of the Tests you wish to archive (Figure 16 - 1).

Figure 16 - Bulk archiveImage Added

Figure 16 - Bulk archive

UI Step

Once you've checked the boxes, the Selected button (Figure 16 - 2) will appear. Click it and you will see a dropdown menu. There, click Archive (Figure 16 - 3).

UI Step

A modal will open for you to confirm this operation.

Click Archive (Figure 17 - 1) to proceed.

Figure 17 - ArchiveImage Added

Figure 17 - Archive

Viewing Archived Test Runs

UI Steps
UI Step

On the Test Issue, click the ellipsis button (Figure 18 - 1) and then click Show archived... (Figure 18 - 2).

Figure 18 - ArchivedImage Added

Figure 18 - Archived

UI Step

A modal will open, showing all archived Test Runs for the Test you are on (Figure 19).

In this modal (Figure 19) you can click the ellipsis button (Figure 19 - 1) and then restore/delete (Figure 19 - 2; 3) that Test Run.

Figure 19 - ArchivedImage Added

Figure 19 - Archived


Info

...

Both of these actions can be performed in bulk by

...

checking the boxes (Figure 19 - 5) and then clicking the restore/delete options (Figure 19 - 3).

UI Step

Click Close (Figure 19 - 4) to go back to the Test Issue (Figure 18).


Info

You can also execute, assign, and remove Test Runs (Figure 16 - 3).


Expand
titleSupport/Troubleshooting

If you have questions or technical issues, please contact the Support team via the Customer Portal (Jira service management) or send us a message using the in-app chat.

Step 1: Selecting the archived test runs by clicking on the checkbox at the left of each test run.

Step 2: Click on the bottom that appears at the top of the table and then the desired action.

Step 3: Confirm the action.

Image Removed

Image Removed

...