What you'll learn

  • How to export test cases from TestRail
  • Use a script to convert the exported XML test cases to an Xray compatible CSV file
  • Import different test cases (with different fields and options)
  • Validate that the test cases were imported to Xray

Source-code for this tutorial

  • All examples are available on GitHub

Overview

This tutorial explains how to export test cases from TestRail and import them into Xray.

TestRail is a test management tool that runs independently with versions available on-premises and in the cloud. In this tutorial, you will learn how to export the test cases from TestRail in an XML format, how to use a script (available in GitHub) to convert the XML file into a compatible CSV so that it can be imported into Xray using the Test Case Importer.

This tutorial will provide different examples of exporting and importing test cases from TestRail into Xray. 

Features and Limitations

Below is a list of the support features and current limitations.

Migration from TestRail is limited to the information we can export from TestRail and how it can be mapped to Xray entities.

Most of the test case information will be migrated seamlessly, but please check the following table for more details.

Supported FeaturesUnsupported
  • Test and Test steps
  • Preconditions
  • Test Sections/SubSections
  • Test case attachments
  • Test Runs
  • Test Plans
  • Defects and Requirements


How it works

TestRail allows users to create different test projects. The user can specify and organize those tests into test sections. Each test project can have multiple test sections and subsections.

TestRail has a set of templates we choose upon test case creation that will determine the information in the test. The default templates have a "Test Case (Text)" that allows you to describe the test case in a text format, and we also have a "Test Case (Steps)" that allows you to describe a test case in different steps with action, additional data and expected result. TestRail also has a Behaviour Driven Development template that enables the user to describe the test into different scenarios or an Exploratory test type that defines a mission and goals for the test.

In TestRail, to execute tests, you need to associate them to a test run.

TestRail uses different entities to describe tests and the flow, so we have synthesized into the next table the existing mappings between TestRail and Xray.

TestRailXrayComments
Test CaseTest

TestRail does not support data sets or Gherkin definition; only manual/automated test steps.

In TestRail when you choose to export using XML format it will export everything. All steps are exported unless they are empty.

Precondition

(Field in Test Case entity)

Precondition

(independent entity)

In TestRail, the precondition is one field in the test case.

In Xray, a Precondition is a different entity and one Test Case can be linked to several Preconditions.

The precondition field from TestRail is converted into a Precondition issue in Xray and linked back to the Test.

Test SectionTest Repository

The test sections and sub-sections of TestRail are mapped into Xray Test Repository.  



Prerequisites


For these examples we will use the Test Case Importer tool, which is present when you install Xray in your Jira instance.

 We will need:

  • Access to TestRail instance to export test cases in XML format
  • Access and usage of scripts to convert TestRail XML test cases to CSV Xray import files
  • Access to a Jira instance with Xray installed

  • Only users that have the Make Bulk Changes permission will have this option available. Additionally, the user will only be able to import data into projects where he has the Create Issues permission.



Examples

In each example, we will show how we obtained the XML file from TestRail, made the files available in GitHub, how to use the scripts to convert the XML files into compatible CSV files, and finally, how to import the CSV files into Xray.

We will showcase different possibilities when exporting test cases from TestRail, using a combination of fields and possibilities that should cover most usages.

All of the examples in this tutorial have an XML file exported from TestRail, a CSV file with the definition of the test cases to import, and a configuration file that will configure all associations and fields for the importation to be successful.


Export to XML from TestRail and Import into Xray

In TestRail, when you choose to export using XML format, it will export everything; no option is given to choose to specify what you want to export. In this example, we are focusing on that option.

To export from TestRail, we are using the Export to XML option in the Test Case upper menu entry.


We provide a name for the XML field that will be exported and click "Export".

Looking into more detail on what we have in TestRail, we can see that we have defined 3 sections:

  • Test Cases (with one sub-section)
    • Regression
  • Exploratory Tests
  • Performance Tests


The file generated by TestRail has the following content:

comic_estore.xml 
<?xml version="1.0" encoding="UTF-8"?>
<suite>
	<id>S196</id>
	<name>Master</name>
	<description></description>
	<sections>
		<section>
		<name>Test Cases</name>
		<description></description>
						<cases>
												<case>
				<id>C22478</id>
				<title>Test Case Steps (Text)</title>
												<template>Test Case (Text)</template>
																<type>Destructive</type>
																<priority>Medium</priority>
																				<estimate></estimate>
																								<references></references>
																				<custom>
										<automation_type><id>6</id><value> Manual</value></automation_type>
										<preconds>Test Case Step *TEXT* Precondition</preconds>
										<steps>%Var3 
%Var4
</steps>
										<expected> ![](index.php?/attachments/get/d8d1d9e0-fe4c-43c8-b894-dfd83d9c160c) </expected>
									</custom>
							</case>
															<case>
				<id>C22494</id>
				<title>Automated Checkout</title>
												<template>functional area</template>
																<type>Automated</type>
																<priority>Critical</priority>
																				<estimate></estimate>
																								<references></references>
																			</case>
								</cases>
								<sections>
		<section>
		<name>Regression</name>
		<description>Regression Tests</description>
						<cases>
												<case>
				<id>C22475</id>
				<title>Manual Test Case (Steps)</title>
												<template>Test Case (Steps)</template>
																<type>UAT</type>
																<priority>Medium</priority>
																				<estimate></estimate>
																								<references></references>
																				<custom>
										<automation_type><id>6</id><value> Manual</value></automation_type>
										<preconds>Manual Test Case (Steps) Precondition</preconds>
										<steps_separated><step>
<index>1</index><content>Login with user *%User1* 
Expected Result:
Application redirects you to the main page in a logged in state</content></step>
<step>
<index>2</index><content>First step description</content><expected>Expected result for the first step</expected><additional_info>Additional step description</additional_info></step>
<step>
<index>3</index><content>Second step description with two variables: %Var1% and %Var2% </content><expected>Expected result for the second step</expected><additional_info>Additional step 2 description</additional_info></step>
<step>
<index>4</index><content>Third step with a table:




|||:Name 1|:Last name 2|:Phone 3|:Address 4
|| John .. | Smack | 111111111 |  Address1
|| Smith .. | Moon |222222222  |  Address2
|| Jenkins .. | Stone | 333333333 |  Address3
|| Ford .. | Mustang |444444444  |  Address4
</content><expected>Expected result for the third step</expected><additional_info>Additional step 2 description (Var1: %Var1% and Var2: %Var2%)</additional_info></step>
<step>
<index>5</index><content> ![](index.php?/attachments/get/3b93e38b-5609-49c6-a987-8f2f6968121e) </content><expected> ![](index.php?/attachments/get/6c9ccdd6-446f-4b23-9244-3a16b997a0e6) </expected><additional_info> ![](index.php?/attachments/get/5edccf31-330d-4e62-9e8a-324f771d96c8) </additional_info></step>
</steps_separated>
									</custom>
							</case>
								</cases>
							</section>
	</sections>
			</section>
			<section>
		<name>Exploratory Tests</name>
		<description>Exploratory Test Section Description</description>
						<cases>
												<case>
				<id>C22477</id>
				<title>Exploratory Session Test</title>
												<template>Exploratory Session</template>
																<type>Regression</type>
																<priority>Medium</priority>
								<assignedto>Cristiano Cunha</assignedto>
												<estimate></estimate>
																								<references></references>
																				<custom>
										<mission>Exploratory Mission</mission>
										<goals>Exploratory Goal</goals>
									</custom>
							</case>
								</cases>
							</section>
			<section>
		<name>Performance Tests</name>
		<description></description>
						<cases>
												<case>
				<id>C22493</id>
				<title>Load Tests</title>
												<template>functional area</template>
																<type>Performance</type>
																<priority>Medium</priority>
																				<estimate></estimate>
																								<references></references>
																			</case>
								</cases>
							</section>
	</sections>
</suite>


To import into Xray, we created one script to convert this XML file into a CSV-compatible file to be imported into Xray.

The script will parse the XML file and create a CSV file that can be imported into Xray. To execute the script, use the following command:

python3 testrail2Xray.py -i comic_estore.xml -o comic_estore.csv


The output file has all the test steps and, if you have used a precondition in TestRail, it will create a precondition in Xray and link it to the test.

comic_estore.csv 
Issue ID,Issue Key,Test Type,Test Summary,Test Priority,Action,Data,Result,Test Repo,Precondition,Issue Type,Precondition Type,Unstructured Definition,Labels
1,,,Test Case Step *TEXT* Precondition,,,,,,,precondition,Manual,,
2,,Manual,Test Case Steps (Text),2,"%Var3 
%Var4
",, ![](index.php?/attachments/get/d8d1d9e0-fe4c-43c8-b894-dfd83d9c160c) ,Test Cases,1,,,,Destructive
3,,Manual,Automated Checkout,3,,,,Test Cases,,,,,
4,,,Manual Test Case (Steps) Precondition,,,,,,,precondition,Manual,,
5,,Manual,Manual Test Case (Steps),2,"Login with user *%User1* 
Expected Result:
Application redirects you to the main page in a logged in state",,,Test Cases/Regression,4,,,,UAT
5,,Manual,,3,First step description,Additional step description,Expected result for the first step,,,,,,
5,,Manual,,3,Second step description with two variables: %Var1% and %Var2% ,Additional step 2 description,Expected result for the second step,,,,,,
5,,Manual,,3,"Third step with a table:




|||:Name 1|:Last name 2|:Phone 3|:Address 4
|| John .. | Smack | 111111111 |  Address1
|| Smith .. | Moon |222222222  |  Address2
|| Jenkins .. | Stone | 333333333 |  Address3
|| Ford .. | Mustang |444444444  |  Address4
",Additional step 2 description (Var1: %Var1% and Var2: %Var2%),Expected result for the third step,,,,,,
5,,Manual,,3, ![](index.php?/attachments/get/3b93e38b-5609-49c6-a987-8f2f6968121e) , ![](index.php?/attachments/get/5edccf31-330d-4e62-9e8a-324f771d96c8) , ![](index.php?/attachments/get/6c9ccdd6-446f-4b23-9244-3a16b997a0e6) ,,,,,,
6,,Exploratory,Exploratory Session Test,2,,,,Exploratory Tests,,,,"*Mission:* Exploratory Mission
 *Goals:* Exploratory Goal
",Regression
7,,Manual,Load Tests,2,,,,Performance Tests,,,,,


To import it into Xray, we use Test Case Importer with the recently created CSV file; you can use the configuration file provided with the .json extension (importConfiguration.json).

Once imported we can see that it has created two Preconditions and five Tests with all the properties defined in the CSV.


More Details

Let's look into some specificities of this approach and some test particularities.

precondition field into Precondition issue

In TestRail Preconditions are another field in the Test.


The precondition is exported in a 'preconds' field within 'custom' as we can see below. 

precondition in comic_estore.xml 
...
<custom>
	<automation_type><id>6</id><value> Manual</value></automation_type>
	<preconds>Test Case Step *TEXT* Precondition</preconds>
	<steps>%Var3 
%Var4
	</steps>
	<expected> ![](index.php?/attachments/get/d8d1d9e0-fe4c-43c8-b894-dfd83d9c160c) </expected>
</custom>
...


When converting into a CSV for Xray we are creating a new issue of type 'precondition' and linking it to the Test.

Precondition in comic_estore.csv 
Issue ID,Issue Key,Test Type,Test Summary,Test Priority,Action,Data,Result,Test Repo,Precondition,Issue Type,Precondition Type,Unstructured Definition,Labels
1,,,Test Case Step *TEXT* Precondition,,,,,,,precondition,Manual,,
2,,Manual,Test Case Steps (Text),2,"%Var3 
%Var4
",, ![](index.php?/attachments/get/d8d1d9e0-fe4c-43c8-b894-dfd83d9c160c) ,Test Cases,1,,,,Destructive
...


Once imported into Xray, we can see that the first line creates a Precondition in Xray.

The next line creates a Test and links the Precondition created in the same operation.

Test Sections into Test Repository

In TestRail, we can organize our tests into sections and subsections as we can see below:

As we can see in the above picture we have:

  • one section called 'Test Cases' with 2 tests and a sub-section 'Regression' with one test
  • one section called 'Exploratory Tests' with one test
  • one section called 'Performance Tests' with one test

TestRail sections and sub-sections are exported into the XML file in the following form:

test section in comic_estore.xml 
<?xml version="1.0" encoding="UTF-8"?>
<suite>
	<id>S196</id>
	<name>Master</name>
	<description></description>
	<sections>
		<section>
			<name>Test Cases</name>
			<description></description>
						...
			<sections>
				<section>
					<name>Regression</name>
					<description>Regression Tests</description>
					...
				</section>
			</sections>
		</section>
		<section>
			<name>Exploratory Tests</name>
			<description>Exploratory Test Section Description</description>
						<cases>
							...
						</cases>
		</section>
		<section>
			<name>Performance Tests</name>
			<description></description>
						<cases>
							<case>
								...
		</section>
	</sections>
</suite>


After running the script that converts the XML file into a CSV format we can see that it adds, in the Test Repo field, the Test Repository name to be created when uploading the file into Xray.

Test Repo in comic_estore.csv 
Issue ID,Issue Key,Test Type,Test Summary,Test Priority,Action,Data,Result,Test Repo,Precondition,Issue Type,Precondition Type,Unstructured Definition,Labels
1,,,Test Case Step *TEXT* Precondition,,,,,,,precondition,Manual,,
2,,Manual,Test Case Steps (Text),2,"%Var3 
%Var4
",, ![](index.php?/attachments/get/d8d1d9e0-fe4c-43c8-b894-dfd83d9c160c) ,Test Cases,1,,,,Destructive
3,,Manual,Automated Checkout,3,,,,Test Cases,,,,,
4,,,Manual Test Case (Steps) Precondition,,,,,,,precondition,Manual,,
5,,Manual,Manual Test Case (Steps),2,"Login with user *%User1* 
Expected Result:
Application redirects you to the main page in a logged in state",,,Test Cases/Regression,4,,,,UAT
...
6,,Exploratory,Exploratory Session Test,2,,,,Exploratory Tests,,,,"*Mission:* Exploratory Mission
 *Goals:* Exploratory Goal
",Regression
7,,Manual,Load Tests,2,,,,Performance Tests,,,,,


After importing into Xray, we can see that new directories were created in the Test Repository:

Manual Tests (With Steps separated and in Text) into Manual Test Case

The templates available by default in TestRail allow you to create Manual Test Cases with steps (using action, data, and expected result) or use only a text field to describe the steps.

We have created one test case using the steps in a text field and one using the steps separated into fields. The XML output is the following:

Manual Tests in comic_estore.xml 
<?xml version="1.0" encoding="UTF-8"?>
<suite>
	<id>S196</id>
	<name>Master</name>
	<description></description>
	<sections>
		<section>
		<name>Test Cases</name>
		<description></description>
			<cases>
				<case>
					<id>C22478</id>
					<title>Test Case Steps (Text)</title>
					<template>Test Case (Text)</template>
					<type>Destructive</type>
					<priority>Medium</priority>
					<estimate></estimate>
					<references></references>
					<custom>
						<automation_type><id>6</id><value> Manual</value></automation_type>
						<preconds>Test Case Step *TEXT* Precondition</preconds>
						<steps>%Var3 
%Var4
						</steps>
						<expected> ![](index.php?/attachments/get/d8d1d9e0-fe4c-43c8-b894-dfd83d9c160c) </expected>
					</custom>
				</case>
				...								
			</cases>
		<sections>
		<section>
		<name>Regression</name>
		<description>Regression Tests</description>
			<cases>
				<case>
					<id>C22475</id>
					<title>Manual Test Case (Steps)</title>
					<template>Test Case (Steps)</template>
					<type>UAT</type>
					<priority>Medium</priority>
					<estimate></estimate>
					<references></references>
					<custom>
						<automation_type><id>6</id><value> Manual</value></automation_type>
						<preconds>Manual Test Case (Steps) Precondition</preconds>
						<steps_separated>
							<step>
								<index>1</index>
								<content>Login with user *%User1* 
Expected Result:
Application redirects you to the main page in a logged in state</content>
							</step>
							<step>
								<index>2</index>
								<content>First step description</content>
								<expected>Expected result for the first step</expected>
								<additional_info>Additional step description</additional_info>
							</step>
							<step>
								<index>3</index>
								<content>Second step description with two variables: %Var1% and %Var2% </content>
								<expected>Expected result for the second step</expected>
								<additional_info>Additional step 2 description</additional_info>
							</step>
							<step>
								<index>4</index>
								<content>Third step with a table:




|||:Name 1|:Last name 2|:Phone 3|:Address 4
|| John .. | Smack | 111111111 |  Address1
|| Smith .. | Moon |222222222  |  Address2
|| Jenkins .. | Stone | 333333333 |  Address3
|| Ford .. | Mustang |444444444  |  Address4
								</content>
								<expected>Expected result for the third step</expected>
								<additional_info>Additional step 2 description (Var1: %Var1% and Var2: %Var2%)</additional_info>
							</step>
							<step>
								<index>5</index>
								<content> ![](index.php?/attachments/get/3b93e38b-5609-49c6-a987-8f2f6968121e) </content>
								<expected> ![](index.php?/attachments/get/6c9ccdd6-446f-4b23-9244-3a16b997a0e6) </expected>
								<additional_info> ![](index.php?/attachments/get/5edccf31-330d-4e62-9e8a-324f771d96c8) </additional_info>
							</step>
						</steps_separated>
					</custom>
				</case>
			</cases>
		</section>
	</sections>
			...


After running the script that converts this XML into a CSV file we can see that, as explained above, the Preconditions issues are created and linked back to the Tests. Each Test is created and the steps filled with the information extracted from the XML.

Manual Tests in comic_estore.csv 
Issue ID,Issue Key,Test Type,Test Summary,Test Priority,Action,Data,Result,Test Repo,Precondition,Issue Type,Precondition Type,Unstructured Definition,Labels
1,,,Test Case Step *TEXT* Precondition,,,,,,,precondition,Manual,,
2,,Manual,Test Case Steps (Text),2,"%Var3 
%Var4
",, ![](index.php?/attachments/get/d8d1d9e0-fe4c-43c8-b894-dfd83d9c160c) ,Test Cases,1,,,,Destructive
...
4,,,Manual Test Case (Steps) Precondition,,,,,,,precondition,Manual,,
5,,Manual,Manual Test Case (Steps),2,"Login with user *%User1* 
Expected Result:
Application redirects you to the main page in a logged in state",,,Test Cases/Regression,4,,,,UAT
5,,Manual,,3,First step description,Additional step description,Expected result for the first step,,,,,,
5,,Manual,,3,Second step description with two variables: %Var1% and %Var2% ,Additional step 2 description,Expected result for the second step,,,,,,
5,,Manual,,3,"Third step with a table:




|||:Name 1|:Last name 2|:Phone 3|:Address 4
|| John .. | Smack | 111111111 |  Address1
|| Smith .. | Moon |222222222  |  Address2
|| Jenkins .. | Stone | 333333333 |  Address3
|| Ford .. | Mustang |444444444  |  Address4
",Additional step 2 description (Var1: %Var1% and Var2: %Var2%),Expected result for the third step,,,,,,
5,,Manual,,3, ![](index.php?/attachments/get/3b93e38b-5609-49c6-a987-8f2f6968121e) , ![](index.php?/attachments/get/5edccf31-330d-4e62-9e8a-324f771d96c8) , ![](index.php?/attachments/get/6c9ccdd6-446f-4b23-9244-3a16b997a0e6) ,,,,,,
...


The first Test corresponds to a Manual Test Case with the summary obtained from the CSV, one step with Action and Expected Result, and the added type (Destructive) as a Label.

Notice that we have added one image in the Expected Result in TestRail, and the exported link is missing the TestRail endpoint. To obtain the full URL of the attachment, you need to add your TestRail endpoint to the start of the image's link; using the endpoint obtained from concatenating the TestRail endpoint and the attachment URL, you can access it.

The second Test has one line per step. Notice that the format is kept for the table and styles added in TestRail.

Exploratory Tests into Exploratory Test Case

TestRail also allows the creation of Exploratory Tests, this type of Test is composed of a mission and goals as we can see in the exported XML file below.

Exploratory Test in comic_estore.xml 
<?xml version="1.0" encoding="UTF-8"?>
<suite>
	<id>S196</id>
	<name>Master</name>
	<description></description>
	<sections>
	...
		<section>
			<name>Exploratory Tests</name>
			<description>Exploratory Test Section Description</description>
			<cases>
				<case>
					<id>C22477</id>
					<title>Exploratory Session Test</title>
					<template>Exploratory Session</template>
					<type>Regression</type>
					<priority>Medium</priority>
					<assignedto>Cristiano Cunha</assignedto>
					<estimate></estimate>
					<references></references>
					<custom>
						<mission>Exploratory Mission</mission>
						<goals>Exploratory Goal</goals>
					</custom>
				</case>
			</cases>
		</section>
			....


To have this working, make sure you have an Exploratory Test Type added to your instance; if not, please follow this tutorial to do so.

After converting the above file using the available script, we get the following output

Exploratory Test in comic_estore.csv 
Issue ID,Issue Key,Test Type,Test Summary,Test Priority,Action,Data,Result,Test Repo,Precondition,Issue Type,Precondition Type,Unstructured Definition,Labels
...
6,,Exploratory,Exploratory Session Test,2,,,,Exploratory Tests,,,,"*Mission:* Exploratory Mission
 *Goals:* Exploratory Goal
",Regression
...


When importing into Xray it will create an Exploratory Test and add the Goals and Mission into the Description field.

Automated Tests into Automated Test Cases

The last type supported is the Automated type in TestRail, this test has no description due to its nature.

The exported XML looks like the one below:

Automated Test in comic_estore.xml 
<?xml version="1.0" encoding="UTF-8"?>
<suite>
	<id>S196</id>
	<name>Master</name>
	<description></description>
	<sections>
		<section>
			<name>Test Cases</name>
			<description></description>
			<cases>
				...
				<case>
					<id>C22494</id>
					<title>Automated Checkout</title>
					<template>functional area</template>
					<type>Automated</type>
					<priority>Critical</priority>
					<estimate></estimate>
					<references></references>
				</case>
			</cases>
			...
	</sections>
</suite>


After using the script provided it will convert the XML into the following CSV file.

Automated Test in comic_Estore.csv 
Issue ID,Issue Key,Test Type,Test Summary,Test Priority,Action,Data,Result,Test Repo,Precondition,Issue Type,Precondition Type,Unstructured Definition,Labels
...
3,,Generic,Automated Checkout,1,,,,Test Cases,,,,,Automated
...


The script has converted the TestRail Automated type into the Xray Generic type, added it to the Test Cases  directory in the Test Repository and added the Label 'Automated' to it.

Other Tests

The script will convert all tests of type Manual, Exploratory or Automated; all the other types will be discarded and no output is created in the resulting CSV file.

The Type field (Destructive, Smoke, UAT, etc) in TestRail is converted into labels and added in Xray.



This tutorial does not cover BDD Test Cases due to a TestRail limitation.



Tips

  • Use the CSV file provided to import the examples and the Json configuration associated to have the setup and mapping correct.
  • Ensure you are using the scripts available for the right version of Jira (Cloud or Server).