6 September 2021

The Xray team is proud to announce the release of Xray 5.1.

This version's features include new capabilities for importing execution results using the Xray JSON format and the JUnit XML format, supporting the Actual Results field when creating defects and improvements on datasets.

Upgrade Notes

Upgrading Xray to 5

If you are upgrading from a version lower than 3.2, this implies that an upgrade task will run to pre-calculate historical data. This is a one-time action that will run in the background so that Xray is still available during the upgrade. A  re-index operation of your Jira instance is required so that the internal data becomes searchable and usable by Xray.

Please be aware that this upgrade task will run immediately after updating Xray. As this operation can take some time and use a considerable amount of server resources, we suggest scheduling the upgrade of Xray outside of business hours. It is always possible to cancel the recalculation operation in the Xray configuration section → Custom Fields → Maintenance.

If you are upgrading from v3.2.x or v3.3.x to v5, you must recalculate data in the Xray configuration section → Custom Fields. This recalculation is necessary because of the huge improvement in the amount of data that Xray needs to add to the table and to the Lucene indexes.

As this operation can take some time and use a considerable amount of server resources, we suggest scheduling the upgrade of Xray and the recalculation of the custom fields, outside of business hours.

Downgrading from Xray 5

Downgrading from Xray 5.x.x to a previous version will have the following side-effects on Test Runs with a dataset:

  • Test parameters will no longer be converted on the Test Run page, this includes Test Step definitions and Pre-Conditions;
  • When loading a Test Run created on version 5.x.x, the merge/reset message will be displayed;
  • When merging a Test Run created on version 5.x.x with the new Test Step definition, all Test Step results will be updated but the reset/merge message will still be displayed;
  • A Test Run created on version 5.x.x with iterations will be displayed in a "flat mode." The iterations will no longer be available and all the Test Step results from each iteration will be presented on the Test Steps section;
  • After the downgrade, the integrity checker will fail on "Check for invalid Mementos" for all Test Runs created on version 5.x.x. We recommend running the integrity checker before downgrading to get rid of possible invalid mementos.

The Test Step Custom Fields and Test Run Custom Fields of the type "User Picker" will have to be deleted in the project configuration as they are not supported. Following that, the Test Runs that are using a User Picker field will have to be merged or reset. This includes Test Runs that have the field for the Test Run itself and also Test Runs for Manual Tests where the Test Steps contained the User Picker field.

Downgrading from Xray 5.x to a version less than 3.2.0 implies that you must perform a re-index afterward. This is because from Xray 3.2.0 onwards, the content and format of what Xray indexes have changed.

Historical Coverage Gadget

Some configuration parameters for this gadget were modified as a result of supporting multiple projects whenever using a saved filter as a source. It is possible that some users will get errors on the gadget after downgrading from 4.x. If this happens, please edit the configuration to ensure the gadget is configured properly. 

Release highlights 

New Xray JSON format capabilities

When importing execution results using the Xray JSON format, you can create tests automatically by providing a testInfo element along with the test specification. Xray will only create the tests once since consequent imports of the same report will reuse the same test issues and generate new executions for the results. Manual and BDD tests are matched by summary and Generic tests by the generic definition within the same project.

Xray brings you the ability to import data-driven test results into manual tests. A dataset within the execution will automatically be created based on the iterations that are being imported.

Learn more about this feature here.

JUnit extension 

We have created an extension for JUnit in order to provide much-needed information to Xray when importing execution results such as:

  • tracking started and finished date timestamps for each test
  • linking a test method to an existing Test issue or use auto-provisioning
  • covering a "requirement" (i.e. an issue in Jira) from a test method
  • specifying additional fields for the auto-provisioned Test issues (e.g. summary, description, labels)
  • attaching screenshots or any other file as evidence to the Test Run, right from within the test method
  • adding comments on the Test Run, right from within the test method
  • setting the values for Test Run custom fields, right from within the test method

You can find the extension on our Github page with instructions on how to install and use it.

Learn more here.

The actual results field will now be included when creating defects

When creating defects for manual tests, Xray will now include the actual results field on the defect issue description as a new column for each step.

This will include any embedded images within the actual results field.

Learn more here.

Dataset improvements

Xray is now able to export existing datasets into CSV. This will allow you to edit the datasets outside of Jira and import them back again.

You can also change the order of parameters within a dataset.

The default size for the parameter values has increased to 100 chars. You can configure this size through the Xray global settings up to a maximum of 255 chars.

Learn more about parameterized tests here.

And more...

UI/UX Improvements

  • XRAY-7871: UI Improvements to Web panel toolbars
  • XRAY-7876As a user, I can create a new Test Execution issue for a given requirement issue


  • XRAY-7480: Export Execution Results - REST API should include Test Run Custom Fields results
  • XRAY-7868As a REST API user, I can export the dataset associated with a test in a given context


  • XRAY-2528: Xray must support multiple examples tables for Cucumber Scenario outline Tests

All New Features and Bug Fixes in this release


Key Summary T Created P Status Resolution

  • No labels