Switching test management tools can be cumbersome, with the overhead of moving your assets and relearning the processes. But do not fret, with some help, you can take the steps necessary to streamline the experience significantly.
In this tutorial, we will provide guidance for users migrating from Zephyr Scale to Xray in Jira Cloud, focusing on the testing workflow and best practices.
Note: many of the concepts and flows are similar (if not exactly the same) in Zephyr Scale for Data Center, so this tutorial could be used as high-level guidance for the migration from there to Xray Cloud as well.
Before you migrate
To ensure your data is not lost in transition, you first need to identify all the migration steps. The preferred sequence is:
- Evaluate whether any adjustments are needed for Zephyr test cases given their XML export limitations and the desired Xray layout & format (see the Mapping section below).
- Export the Zephyr test case data - follow the initial steps in this tutorial.
- Disable Zephyr Scale - uninstalling the plugin prevents you from accessing any Zephyr-specific functionality (test step data, test cycle and test execution details).
- If you must have Zephyr Scale and Xray plugins active simultaneously, make sure they don’t share the “Test” issue type name (i.e. “Test Case” or “TestZephyr” is ok).
- Enable Xray and consider if any elements need to be created/configured before the import of test cases (see the Mapping section below).
- Prepare the test cases from step 2 accordingly and import them to Xray following the rest of the tutorial.
Mapping
As you are migrating from one Jira Cloud plugin to the other, the support for Jira’s native fields will be similar - you can assume that the elements we don’t explicitly mention are consistent between the plugins.
So, we will focus our tutorial on the plugin-specific concepts and entities. At the project level:
Zephyr Scale | Xray | Comments |
Zephyr Scale link in the left navigation menu | Testing Board link in the left navigation menu | Xray also supports cross-project operations, you have quite a lot of flexibility in structuring you assets (see more here). Xray entities are native Jira issues, so you can apply operations like JQL, workflows for status transitions, Jira Automation to them. And as a side note, Xray and Xray Enterprise are also Cloud Fortified. |
Test Cases tab | Test Repository Folders/Subfolders | You can move the test cases after the import or create the folder structure in advance then map it during the import. You can also consider using Test Sets. |
Test Case | Test | Xray’s Test layout is more similar to the one from standard Jira issue types. Xray supports all types of tests from Zephyr Scale as well as custom ones. Related features that Xray supports:
|
Test Cycle | Test Execution issue | Similar to Zephyr Scale entities, Xray Test Execution is mandatory for running the tests while Xray Test Plan serves as an optional higher-level layer, especially useful when the same group of tests is run in multiple Test Executions. The order of creation in Xray does not matter (e.g. you can create an Xray Test Plan first, Xray Test Execution second). You can add tests directly to the Xray Test Plan. You can have more than 1 Xray Test Execution per Sprint. You can clone both Xray Test Executions and Xray Test Plans. Xray also supports easy re-running of tests by status. |
Test Plan | Test Plan issue | |
Test Player inside Test Cycle A line item on the Executions tab in the Test Case view | Test Run Details | In Xray Test Execution issues, you click the “Play” button for each test individually, then the aspects like step-level results or attachments are handled in the Test Run Details view. |
Test Cycle environment | Multiple Test Execution issues with different Test Environments | The use of Test Environments in Xray is very broad. This is also a good use case for a Test Plan linked to those multiple Test Executions. |
Reports tab | Reports section of the Testing Board | Both plugins provide traceability, coverage, and execution-focused reports. In addition, Xray provides a more comprehensive list of scopes (e.g. requirements, test plans). Also, you can leverage Document Generator or Xporter for additional insights. Xray Dashboard gadgets are also available out of the box. |
Configuration | Project Settings -> Apps -> Xray Settings Manage Apps -> Xray | |
You can find some of the integration and automation examples in these sections of Xray documentation:
|
At the Story and Test level:
Zephyr Scale | Xray | Comments |
Zephyr Scale button at the top of the Story and Zephyr Scale section | Test Coverage button at the top of the Story and Test Coverage section | When a test is created from the Story screen, traceability is established automatically in Xray as well. You can link Xray Test Executions and Test Plans to the Story or see the aggregated execution status in the Test Coverage section. You can learn more about Xray coverage calculations in this article. |
Plain Text type of Test Case | Generic type of Test | Keep in mind - Zephyr Scale does not export the test case attachments in the XML file. |
Step-by-Step type of Test Case | Manual type of Test | |
BDD-Gherkin type of Test Case | Cucumber type of Test | |
Details tab | Jira’s Description and other metadata fields | For Xray, this information is handled in Jira’s settings for issue layouts. I.e. if you are using custom fields in Zephyr, you need to create them for Jira issues and apply that layout to Xray tests. A few elements to highlight:
|
Precondition (Field in Test Case entity) | Precondition (independent entity) | In Zephyr Scale, the precondition is one field in the test case. In Xray, a Precondition is a different entity and one Test Case can be linked to several Preconditions. If you follow the instructions in the import tutorial, the precondition field from Zephyr Scale is converted into a Precondition issue in Xray and linked back to the Test. |
Test Script tab | Test Details section | The layout for all 3 test types is similar between the plugins. You can also create custom step-level fields in Xray settings. |
Executions tab | Test Runs section | |
Traceability tab | Linked issues and Web Links sections Test Coverage section | The way Jira “sees” the links is a bit different between the plugins, but you should be able to map the “Coverage”/”Links” column from the Zephyr export to the “Outward link (tests)” one in the Xray import. |
Attachments tab | Attach button in the Jira issue layout | Creating attachments at the test case level and the test step one is supported by both plugins. |
Comments and History tabs | Comments and History subsections under the Activity section |
To summarize, for the migration, you will need to pay particular attention to:
- ways of organizing your tests (i.e. recreating folders, test executions, and possibly test plans/test sets);
- attachments (move them to non-Zephyr issues like associated stories/bugs or transfer to a cloud storage outside Jira then manually reattach to Xray entities);
- custom fields and statuses.
To compare the visual Story layout (same color = same function):
Zephyr Scale
Xray
To compare the visual Test layout (same color = same function):
Zephyr Scale
Xray
Getting started with Xray after migration
If you've used Zephyr Scale before, adopting Xray is easy. Your high-level workflow with Zephyr Scale was likely this:
- Enable Zephyr Scale for a given project
- Open a requirement story
- Use the Zephyr Scale section to create new tests (or add existing ones)
- Define the Test Case Details and Test Script
- Create a Test Cycle for execution and add the test(s)
- (Optional) Create a Test Plan to group Test Cycles
- Execute
- See the results on the requirement story
- Utilize the reports and/or dashboards for project-level analysis
In Xray, your actions would be similar:
- Enable Xray for a given project
- Map your issue types to requirements and/or defects to make the coverable by Xray tests (in global settings - Jira Administration > Manage Apps > Xray > Issue Type Mapping)
- Activate Xray Requirement Coverage for the project
- Open a requirement story
- Use the Test Coverage section to create new tests (or add existing ones)
- Define the Test Details and issue metadata fields
- Create an Xray Test Execution issue from the “Test Runs” section of Test Details and add more tests to it, if desired
- (optional) Create an Xray Test Plan issue from the respective section of Test Details to group Xray Test Executions
- Execute
- See the results on the requirement story
- Utilize the reports (built-in and/or from Document Generator/Xporter) and/or dashboards for project-level analysis
You can watch our Xray Cloud product demo for a more comprehensive process walkthrough and check out the Xray Enterprise feature set.
Best practices for Xray adoption
For a consolidated collection of useful tips please refer to this Process section, as well as the “Tutorial, Tips, and Tricks” category overall. Here, we highlight just a few best practices:
- Avoid cloning issues; reuse as much as possible.
- modular and parameterized test, versioning.
- Test Environments.
- You can leverage Jira Automation to make Test issues inherit metadata from the stories.
- Xray can take advantage of Jira’s built-in workflow mechanism in order to provide greater control over the specification, execution and planning.
- Restrict data (i.e., filter it) in the reports or gadgets to make the insights more focused.
- Keep in mind that Xray has both global and project-specific settings.
- Leveraging “Health Check” in global settings every 3 months is recommended.
- You can track storage usage in Xray global settings; we advise you to perform this periodically.
- Remember that JQL and filters are used extensively throughout Jira, so teams should be aware that they should optimize their queries and use proper JQL functions.
- Have control over the REST API as it’s easy to “get excited with it” and abuse it.
- We highly recommend you to take Atlassian University Course Jira Governance & Housekeeping as most of the recommendation do apply to Xray
Xray and its Enterprise version are very comprehensive solutions for test management, so there's much to explore. We recommend checking out the academy courses, product guide, and tutorials.