Switching test management tools can be cumbersome, with the overhead of moving your assets and relearning the processes. But do not fret, with some help, you can take the steps necessary to streamline the experience significantly.
In this tutorial, we will provide guidance for users migrating from Azure DevOps (ADO) Services (specifically, Test Plans) to Xray in Jira Cloud, focusing on the testing workflow and best practices. For the avoidance of doubt, you could still keep using ADO for automation purposes as described in our tutorial for Azure DevOps
Disclaimer: concepts and UI elements may be slightly different in other Azure DevOps versions, but the high-level process should be consistent.
Before you migrate
To ensure your data is not lost in transition, you first need to identify all the migration steps. The preferred sequence is:
- Evaluate whether any adjustments are needed for Azure DevOps Boards entities given the desired Jira layout (see the Mapping section below).
- Migrate the requirements data from Boards (Jira import requirements, Jira REST API) and make any corrections in Jira, if necessary. Options:
- CSV export (most straightforward)
- Office integration
- Azure DevOps REST API (e.g. option 1, option 2)
- Export the test cases from Azure DevOps Test Plans
- CSV export (most straightforward)
- Azure DevOps REST API (e.g. Get Test Suite)
- Enable Xray and consider if any elements need to be created/configured before the import of test cases (see the Mapping section below).
- Prepare the test cases from step 3 and import them to Xray.
- The most noticeable adjustment is deleting the preceding empty cells in the Test Step, Step Action, and Step Expected columns of the CSV exported from ADO.
And as security is one of the highlighted features of Azure DevOps, we wanted to mention that:
- Atlassian is ISO 27001 certified.
- Xray holds a SOC 2 Type 2 certification. Both Xray and Xray Enterprise have been awarded the Atlassian Cloud Fortified Certification.
Mapping
Even though you are migrating from one test management platform to another, some concepts will still be similar (maybe with different labels) as both tools facilitate Agile and DevOps practices with compatible/consistent approaches.
We will focus our tutorial on the major differences. At the project level, for both Jira and Xray:
ADO Test Plans | Jira/Xray | Comments |
Project | Jira Project | You can choose the project template that best suits you (Kanban, Scrum, etc.). Keep in mind the differences between team- and company-managed projects. Also, if desired, you can have requirements in one project and testing entities in another, with cross-project traceability and reporting. |
Iteration | Jira Sprint and Fix Version | Often, both are used in Jira at the same time to reflect the project iterations (with FixVersion representing longer cycles like “release”). Entities in Jira/Xray can have more than 1 FixVersion value. |
Area | Jira Component Jira Custom fields | Jira Components are a common way to have consistent grouping across your asset types (requirements, tests, and defects). |
Work Item | Jira Issue | In Jira, you can
Nested requirements hierarchy, if desired, can be achieved with Jira add-ons like Structure. Xray issue types (e.g. Test, Test Plan) are treated as any other Jira issue. |
Test Plan | Xray Test Plan Xray Test Repository Folder Xray Test Set | In ADO, a strict hierarchy exists - you create a Test Plan and then organize the tests for execution in folders/subfolders called Test Suites. You can execute directly from the test suite entity because test points/runs are auto-generated. In Xray, the hierarchy is more flexible and multi-dimensional. You can create Test Plan issues and organize them in folders, and you can also use the mix of Repository Folders/Test Sets (with differences described in this section) to organize your testing assets. You can easily access the list of all Test Plans via Xray Testing Board. Then you must create separate Test Execution issues to run the tests. Test Executions can contain any selection of tests, not limited to a given Test Plan, Test Set, or Repository Folder. You can assign Test Executions to relevant Test Plan issues (but not directly to a folder of Test Plans). You can then view the execution results on each involved entity (traceable through to the requirement story via links). When it comes to ADO Test Suite types, there is no direct mapping. Xray also offers both static and dynamic options for adding tests to plans/sets/executions. Those options, however, are not restrictive, you can use a mix of them or switch any time (e.g. you can have 1 Test Plan or 1 Test Execution for a query with multiple requirements). |
Test Suite | Xray Test Execution | |
Test Case | Test | See more in the second table below. |
Tags | Labels | |
Work Item State | Jira Workflow Status | |
Query options | Jira Query Language (JQL) | |
Parameters menu section | You can also consider Xray Enterprise Test Case Designer for more complex data-driven use cases. | |
Configuration menu section | Jira Environment Xray Test Environments | Not variable-driven on the Jira/Xray side. |
Runs/Run Explorer menu section | Xray Test Runs List report Xray Testing Board -> Test Executions | See more in the second table below. |
Charts in a Test Plan/Suite Progress Report | Xray Reports and Dashboard Gadgets Jira Dashboards | Xray provides a comprehensive list of analysis goals (e.g. requirements traceability, test plan metrics, test runs summary) in both standalone and dashboard gadget formats. Filters are used to limit the scope, if desired. Also, you can leverage Xporter or Document Generator for additional insights. |
Pipelines integration | Not exactly identical, but Remote Jobs Triggering serves a similar purpose of connecting your test management and automation more seamlessly. Please also see Xray Integrations and Automation Tutorials, specifically the article about Azure DevOps. Consider Jira integrations with code and deployment tools (e.g. GitHub for Jira). | |
Extensions | Jira add-ons from Atlassian marketplace Xray suite of tools | In addition to Xray Standard, you can consider Xray Enterprise, Xporter, and (as a direct alternative to Test & Feedback extension) Xray Exploratory App. |
Note: the mapping above is recommended for better clarity but, in many cases, is not mandatory. E.g. you could map Area to Labels, Test Suites to the combination of Test Plans and Test Executions, etc.
At the tactical level for Xray:
ADO Test Plans | Xray | Comments |
Define tab in a Test Suite | You can define Tests from:
| Xray Test issue layout overall is the same as any other Jira issue. You can use these Jira Automation tips to inherit data from requirement stories. Tests can be copied, shared, assigned, and approved according to Jira permission schemes and workflow statuses. Robust test case versioning is available as part of Xray Enterprise while basic history can be found on the corresponding tabs of an Xray Test issue. From the migration perspective, Supported:
Not supported:
For linked Defects, Requirements, Test Plans/Sets/Repository folders, if you have already migrated those entities to Jira (or recreated), you may be able to substitute the values in the ADO export with Jira issue keys, so that the test issue will be linked upon import. Otherwise, you will need to re-link after the import. Once the test entities are migrated, you will need to recreate any applicable attachments, parameters & datasets, and modular (“call”) tests in Xray. |
Manual test case | Manual test Precondition | |
Automated test case | Generic test | |
Exploratory test case (Test & Feedback extension) | Exploratory test | You get a custom test type (“exploratory”) in Xray and can kick off the execution in Xray Exploratory App directly from the Xray UI. |
Title | Summary | |
Shared Steps | Modular Tests (Call Test) | Xray doesn’t provide the “shared steps” feature but provides an alternative approach to call a test as a step in another test. |
Execute tab in a Test Suite | Test Runs tab of a Test issue Test Execution issue | |
Runner/Test Run/Test Point | Test Run of the Test Execution and Test Run Details screen | Test Run in Xray is not a separate Jira entity, it doesn’t have an ID. Xray Test Run automatically inherits the Test Execution properties and their updates. For Test Case changes, you have a choice - see the Data Consistency section. You can create defects directly from Xray Test Runs as well as from the Jira issue creation dialog. You can also leverage Xray settings to control what bug issues inherit from runs. Built-in diagnostic data collection is not available in Xray at this time, but you can attach any evidence files to the Test Run details. |
Runs Hub | See Comments | No exact mapping. To see the results of multiple Xray Test Runs or Executions at the same time, you can leverage Xray Test Runs List report or Xray Testing Board -> Test Executions view. |
To summarize, for the migration, you will need to pay particular attention to:
- custom issue types, fields, and statuses;
- ways of organizing your tests (i.e. recreating test executions and likely test plans/test sets/test repository folders);
- Attachments, shared steps, and parameter lists (move them to a cloud storage, separated from Azure DevOps entities, then manually reattach to Xray entities).
To compare the visual issue layout for test cases (same color = same function):
Azure DevOps Test Plans
Xray
The purple highlight covers different types and locations of “related or linked” entities in Xray, including the repository folder path and the execution-focused Test Runs tab.
Getting started with Xray after migration
If you've used Azure DevOps Test Plans before, adopting Xray is straightforward. Your high-level workflow with ADO (focusing on the testing aspects) was likely this:
- Create a project. Add iterations/sprints.
- Create a Board with a Backlog to establish the project scope.
- Create Test Plan and Test Suite structure.
- Add Test Cases to Test Suites.
- Execute via Test Runs.
- See the results on the Test Case entities and in the Runs section.
- Utilize Charts and Progress Report for project-level analysis.
In Jira and Xray, your actions would be similar:
- Create a Jira project with the appropriate template. Add Fix Versions and Sprints.
- Consider customizing the Jira workflow with statuses and actions.
- Create Epics and Stories to establish the project scope and assign fix version and sprint values via issue fields.
- Enable Xray for the project
- Map your issue types to requirements and/or defects to make them coverable by Xray tests (in global settings - Jira Administration > Manage Apps > Xray > Issue Type Mapping)
- Activate Xray Requirement Coverage for the project
- Create Test Plan and Test Repository/Test Set structure (optional but recommended).
- Create Test issues and connect them to the requirements via “tests” link type.
- Execute via Test Execution issues and associated Test Runs. Assign the necessary tests via direct selection, JQL, or a mix.
- See the results on the Test Execution, Test Plan, Test, or Requirement entities.
- Utilize the reports (built-in from Xray and/or from Xporter/Document Generator) and/or Jira dashboards with varied gadgets for project-level analysis.
You can watch our Xray Cloud product demo for a more comprehensive process walkthrough.
Best practices for Xray adoption
For a consolidated collection of useful tips please refer to this Process section, as well as the “Tutorial, Tips, and Tricks” category overall. Here, we highlight just a few best practices:
- Avoid cloning issues; reuse as much as possible.
- modular and parameterized tests, versioning.
- Test Environments.
- You can leverage Jira Automation to make Test issues inherit metadata from the stories.
- Xray can take advantage of Jira’s built-in workflow mechanism in order to provide greater control over the specification, execution, and planning.
- Restrict data (i.e., filter it) in the reports or gadgets to make the insights more focused.
- Keep in mind that Xray has both global and project-specific settings.
- Leveraging “Health Check” in global settings every 3 months is recommended.
- You can track storage usage in Xray global settings; we advise you to perform this periodically.
- Remember that JQL and filters are used extensively throughout Jira, so teams should be aware that they should optimize their queries and use proper JQL functions.
- Have control over the REST API as it’s easy to “get excited with it” and abuse it.
- We highly recommend you to take Atlassian University Course Jira Governance & Housekeeping as most of the recommendation do apply to Xray
Xray and its Enterprise version are very comprehensive solutions for test management, so there's much to explore. We recommend checking out the academy courses, product guide, and tutorials.