Versions Compared

Old Version 6

changes.mady.by.user Cristiano Cunha

Saved on

compared with

New Version Current

changes.mady.by.user Mariana Santos

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: minor grammar corrections
Info
titleWhat you'll learn
  • How to export test cases from TestRail
  • Use a script to convert the exported XML test cases to an Xray compatible CSV file
  • Import different test cases (with different fields and options)
  • Validate in Jira that the test cases are availablewere imported to Xray
Note
iconfalse
titleSource-code for this tutorial
typeInfo
  • All examples are available in on GitHub

Overview

This tutorial explains how to export test cases from TestRail and import them into Xray.

TestRail is a test management tool that runs independently with versions for available on-premises and in the cloud. In this tutorial, you will learn how to export the test cases from TestRail in an XML format, how to use a script (available in GitHub) to convert the XML file into a compatible CSV so that it can be imported into Xray using the the Test Case Importer.

In this This tutorial , we will provide different examples of exporting and importing test cases from TestRail into Xray. 

If you need help finding or using the tool, please check more information on the Test Case Importer page.


Features and Limitations

Below you can find is a list of the support features and current limitations.

Migration from TestRail is limited by how much to the information we can export from the tool TestRail and how many similarities we can find with Xrayit can be mapped to Xray entities.

Most of the test case information will be migrated seamlessly, but please check the following table for more detaildetails.

Supported FeaturesUnsupported
  • Test and Test steps
  • Test Sections/SubSections
  • Test case attachments
  • Preconditions
  • Test Runs
  • Test Plans
  • Defects and Requirements


How it works

TestRail allows users to create different test projects. The user can specify and organize those tests into test sections. Each test project can have multiple test test sections and subsections.

TestRail has a set of templates that we choose upon test case creation that will determine what the information is present in the test. The default templates have a include the "Test Case (Text)" template that allows you to describe the test case in a text format and ; we also have a "Test Case (Steps)" that allows you to describe a test case in different steps with action, additional data and expected result. TestRail also has a Behaviour Driven Development template that enables the user to describe the test into different scenarios or an Exploratory test type that defines a mission and goals for the test.

In TestRail, to execute tests, you need to associate them to with a test run.
TestRail uses different entities to describe tests and the flow, so we have synthesized into the next table the existing mappings We have summarized a possible mapping between TestRail and Xray entities that we'll use in this tutorial.

TestRailXrayComments
Test CaseTest

Test Case Test TestRail does not support data sets or Gherkin definition, only manual/automated test steps.

All testIn TestRail, when you choose to export using XML format, it will export everything, no option is given. All steps are imported exported unless they are empty.

Test SectionTest Repository

The test sections and sub-sections of TestRail are mapped into Xray Test Repository.  

Info

Notice that for the Server version of Xray we only create creating the Tests corresponding to the manual test cases in TestRail. Preconditions, if needed, must be create created separately. 



Prerequisites


Expand
For these examples we

We will use the Test Case Importer

tool

 tool for these examples, which is present when you install Xray in your Jira instance.

 We will need:

  • Access to TestRail instance to export test cases in XML format
  • Access and usage Usage of scripts to convert TestRail XML test cases to CSV Xray import files
  • Access to a Jira instance with Xray installed



Examples

In each example, we will show how we obtained the XML file from TestRail, how to use the scripts to convert the XML files into compatible CSV files, and finally, how to import the CSV files into Xray. All files are available in on GitHub.

We will showcase different possibilities when exporting test cases from TestRail, using a combination of fields and possibilities that should cover most usages.

All of the examples in this tutorial have a an XML file exported from TestRail, a CSV file with the definition of the test cases to import, and a configuration file that will configure all associations and fields for the importation to be successful in Xray.


Export to XML from TestRail and Import into Xray

In TestRail, when you choose to export using XML format, it will export everything, ; no option is given to choose to specify what we you want to export. In this example, we are focusing on that option.

To export from TestRail, we are using the use the Export to XML option  option in the Test Case upper menu entry.


We have to provide a name for the XML field that will be exported and click "Export".

Looking into more detail on what we have in TestRail we , we can see that we have defined 3 sections:

  • Test Cases (with one sub-section)
    • Regression
  • Exploratory Tests
  • Performance Tests


The file generated by TestRail has the following content:

Code Block
languagetext
titlecomic_estore.xml
collapsetrue
<?xml version="1.0" encoding="UTF-8"?>
<suite>
	<id>S196</id>
	<name>Master</name>
	<description></description>
	<sections>
		<section>
		<name>Test Cases</name>
		<description></description>
						<cases>
												<case>
				<id>C22478</id>
				<title>Test Case Steps (Text)</title>
												<template>Test Case (Text)</template>
																<type>Destructive</type>
																<priority>Medium</priority>
																				<estimate></estimate>
																								<references></references>
																				<custom>
										<automation_type><id>6</id><value> Manual</value></automation_type>
										<preconds>Test Case Step *TEXT* Precondition</preconds>
										<steps>%Var3 
%Var4
</steps>
										<expected> ![](index.php?/attachments/get/d8d1d9e0-fe4c-43c8-b894-dfd83d9c160c) </expected>
									</custom>
							</case>
															<case>
				<id>C22494</id>
				<title>Automated Checkout</title>
												<template>functional area</template>
																<type>Automated</type>
																<priority>Critical</priority>
																				<estimate></estimate>
																								<references></references>
																			</case>
								</cases>
								<sections>
		<section>
		<name>Regression</name>
		<description>Regression Tests</description>
						<cases>
												<case>
				<id>C22475</id>
				<title>Manual Test Case (Steps)</title>
												<template>Test Case (Steps)</template>
																<type>UAT</type>
																<priority>Medium</priority>
																				<estimate></estimate>
																								<references></references>
																				<custom>
										<automation_type><id>6</id><value> Manual</value></automation_type>
										<preconds>Manual Test Case (Steps) Precondition</preconds>
										<steps_separated><step>
<index>1</index><content>Login with user *%User1* 
Expected Result:
Application redirects you to the main page in a logged in state</content></step>
<step>
<index>2</index><content>First step description</content><expected>Expected result for the first step</expected><additional_info>Additional step description</additional_info></step>
<step>
<index>3</index><content>Second step description with two variables: %Var1% and %Var2% </content><expected>Expected result for the second step</expected><additional_info>Additional step 2 description</additional_info></step>
<step>
<index>4</index><content>Third step with a table:




|||:Name 1|:Last name 2|:Phone 3|:Address 4
|| John .. | Smack | 111111111 |  Address1
|| Smith .. | Moon |222222222  |  Address2
|| Jenkins .. | Stone | 333333333 |  Address3
|| Ford .. | Mustang |444444444  |  Address4
</content><expected>Expected result for the third step</expected><additional_info>Additional step 2 description (Var1: %Var1% and Var2: %Var2%)</additional_info></step>
<step>
<index>5</index><content> ![](index.php?/attachments/get/3b93e38b-5609-49c6-a987-8f2f6968121e) </content><expected> ![](index.php?/attachments/get/6c9ccdd6-446f-4b23-9244-3a16b997a0e6) </expected><additional_info> ![](index.php?/attachments/get/5edccf31-330d-4e62-9e8a-324f771d96c8) </additional_info></step>
</steps_separated>
									</custom>
							</case>
								</cases>
							</section>
	</sections>
			</section>
			<section>
		<name>Exploratory Tests</name>
		<description>Exploratory Test Section Description</description>
						<cases>
												<case>
				<id>C22477</id>
				<title>Exploratory Session Test</title>
												<template>Exploratory Session</template>
																<type>Regression</type>
																<priority>Medium</priority>
								<assignedto>Cristiano Cunha</assignedto>
												<estimate></estimate>
																								<references></references>
																				<custom>
										<mission>Exploratory Mission</mission>
										<goals>Exploratory Goal</goals>
									</custom>
							</case>
								</cases>
							</section>
			<section>
		<name>Performance Tests</name>
		<description></description>
						<cases>
												<case>
				<id>C22493</id>
				<title>Load Tests</title>
												<template>functional area</template>
																<type>Performance</type>
																<priority>Medium</priority>
																				<estimate></estimate>
																								<references></references>
																			</case>
								</cases>
							</section>
	</sections>
</suite>


To import into Xray, we created one script to convert this XML file into a CSV-compatible file to be imported in Xray.

The script will parse the XML file and create a CSV file that can be imported into Xray. To execute the script, use the following command:

Code Block
languagepybash
python3 testrail2Xray.py -i comic_estore.xml -o comic_estore.csv

The output file will only have the Manual Tests, for those all the test steps will be present.


An optional parameter (-e) is also available to complete the endpoints sent in the XML file. Just append the above command with the -e option and send the complete endpoint of your TestRail instance. For example, if my TestRail instance is "https://mytestrail.com/," the command will look like this:

Code Block
languagebash
python3 testrail2Xray.py -i comic_estore.xml -o comic_estore.csv -e https://mytestrail.com/


The output file will only have the Manual Tests, for those all the test steps will be present.

Code Block
language
Code Block
languagetext
titlecomic_estore.csv
collapsetrue
Issue ID,Issue Key,Test Type,Test Summary,Test Priority,Action,Data,Result,Test Repo,Labels
1,,Manual,Test Case Steps (Text),3,"%Var3 
%Var4
",, ![](index.php?/attachments/get/d8d1d9e0-fe4c-43c8-b894-dfd83d9c160c) ,Test Cases,Destructive
2,,Manual,Manual Test Case (Steps),3,"Login with user *%User1* 
Expected Result:
Application redirects you to the main page in a logged in state",,,Test Cases/Regression,UAT
2,,Manual,,3,First step description,Additional step description,Expected result for the first step,,
2,,Manual,,3,Second step description with two variables: %Var1% and %Var2% ,Additional step 2 description,Expected result for the second step,,
2,,Manual,,3,"Third step with a table:




|||:Name 1|:Last name 2|:Phone 3|:Address 4
|| John .. | Smack | 111111111 |  Address1
|| Smith .. | Moon |222222222  |  Address2
|| Jenkins .. | Stone | 333333333 |  Address3
|| Ford .. | Mustang |444444444  |  Address4
",Additional step 2 description (Var1: %Var1% and Var2: %Var2%),Expected result for the third step,,
2,,Manual,,3, ![](index.php?/attachments/get/3b93e38b-5609-49c6-a987-8f2f6968121e) , ![](index.php?/attachments/get/5edccf31-330d-4e62-9e8a-324f771d96c8) , ![](index.php?/attachments/get/6c9ccdd6-446f-4b23-9244-3a16b997a0e6) ,,


To import it into Xray, we use the Test Case Importer feature of Xray with the recently created CSV file and ; you can use the configuration file provided with the extension Json the txt extension (XrayCSVImporter-configuration.txt).

Once imported we can see that it has created two Tests with all the properties defined in the CSV.


More Details

Let's look into some specificities of this approach and some test particularities.

Test Sections into Test Repository

In TestRail, we can organize our tests into sections and subsections, as we can see below:

As we can see in the above picture, we have several sections, but only the ones with manual tests will be created in Xray. The sections we have in TestRail are:

  • one section called 'Test Cases' with 2 tests and a sub-section 'Regression' with one test
  • one section called 'Exploratory Tests' with one test
  • one section called 'Performance Tests' with one test

TestRail sections and sub-sections are exported into the XML file in the following form:

Code Block
languagetext
titletest section in comic_estore.xml
collapsetrue
<?xml version="1.0" encoding="UTF-8"?>
<suite>
	<id>S196</id>
	<name>Master</name>
	<description></description>
	<sections>
		<section>
			<name>Test Cases</name>
			<description></description>
						...
			<sections>
				<section>
					<name>Regression</name>
					<description>Regression Tests</description>
					...
				</section>
			</sections>
		</section>
		<section>
			<name>Exploratory Tests</name>
			<description>Exploratory Test Section Description</description>
						<cases>
							...
						</cases>
		</section>
		<section>
			<name>Performance Tests</name>
			<description></description>
						<cases>
							<case>
								...
		</section>
	</sections>
</suite>


After running the script that converts the XML file into a CSV format, we can see that it adds, in the Test Repo field, the Test Repository name to be created , only for the manual tests , when uploading the file into Xray.

Code Block
languagetext
titleTest Repo in comic_estore.csv
collapsetrue
Issue ID,Issue Key,Test Type,Test Summary,Test Priority,Action,Data,Result,Test Repo,Labels
1,,Manual,Test Case Steps (Text),3,"%Var3 
%Var4
",, ![]([Link|https://testrail101.testrail.io/index.php?/attachments/get/d8d1d9e0-fe4c-43c8-b894-dfd83d9c160c)] ,Test Cases,Destructive
2,,Manual,Manual Test Case (Steps),3,"Login with user *%User1* 
Expected Result:
Application redirects you to the main page in a logged in state",,,Test Cases/Regression,UAT
2,,Manual,,3,First step description,Additional step description,Expected result for the first step,,
2,,Manual,,3,Second step description with two variables: %Var1% and %Var2% ,Additional step 2 description,Expected result for the second step,,
2,,Manual,,3,"Third step with a table:




|||:Name 1|:Last name 2|:Phone 3|:Address 4
|| John .. | Smack | 111111111 |  Address1
|| Smith .. | Moon |222222222  |  Address2
|| Jenkins .. | Stone | 333333333 |  Address3
|| Ford .. | Mustang |444444444  |  Address4
",Additional step 2 description (Var1: %Var1% and Var2: %Var2%),Expected result for the third step,,
2,,Manual,,3, ![]([Link|https://testrail101.testrail.io/index.php?/attachments/get/3b93e38b-5609-49c6-a987-8f2f6968121e)] , ![]([Link|https://testrail101.testrail.io/index.php?/attachments/get/5edccf31-330d-4e62-9e8a-324f771d96c8)] , ![]([Link|https://testrail101.testrail.io/index.php?/attachments/get/6c9ccdd6-446f-4b23-9244-3a16b997a0e6)] ,,


After importing into Xray, we can see that new directories were created in the Test Repository:

Manual Tests (With Steps separated and in Text) into Manual Test Case

The templates available by default in TestRail allow you to create Manual Test Cases with steps (using action, data, and expected result) or using use only a text field to describe the steps.

We have created one test case using the steps in a text field and one using the steps separated into fields, the XML output is the following:

Code Block
languagetext
titleManual Tests in comic_estore.xml
collapsetrue
<?xml version="1.0" encoding="UTF-8"?>
<suite>
	<id>S196</id>
	<name>Master</name>
	<description></description>
	<sections>
		<section>
		<name>Test Cases</name>
		<description></description>
			<cases>
				<case>
					<id>C22478</id>
					<title>Test Case Steps (Text)</title>
					<template>Test Case (Text)</template>
					<type>Destructive</type>
					<priority>Medium</priority>
					<estimate></estimate>
					<references></references>
					<custom>
						<automation_type><id>6</id><value> Manual</value></automation_type>
						<preconds>Test Case Step *TEXT* Precondition</preconds>
						<steps>%Var3 
%Var4
						</steps>
						<expected> ![](index.php?/attachments/get/d8d1d9e0-fe4c-43c8-b894-dfd83d9c160c) </expected>
					</custom>
				</case>
				...								
			</cases>
		<sections>
		<section>
		<name>Regression</name>
		<description>Regression Tests</description>
			<cases>
				<case>
					<id>C22475</id>
					<title>Manual Test Case (Steps)</title>
					<template>Test Case (Steps)</template>
					<type>UAT</type>
					<priority>Medium</priority>
					<estimate></estimate>
					<references></references>
					<custom>
						<automation_type><id>6</id><value> Manual</value></automation_type>
						<preconds>Manual Test Case (Steps) Precondition</preconds>
						<steps_separated>
							<step>
								<index>1</index>
								<content>Login with user *%User1* 
Expected Result:
Application redirects you to the main page in a logged in state</content>
							</step>
							<step>
								<index>2</index>
								<content>First step description</content>
								<expected>Expected result for the first step</expected>
								<additional_info>Additional step description</additional_info>
							</step>
							<step>
								<index>3</index>
								<content>Second step description with two variables: %Var1% and %Var2% </content>
								<expected>Expected result for the second step</expected>
								<additional_info>Additional step 2 description</additional_info>
							</step>
							<step>
								<index>4</index>
								<content>Third step with a table:




|||:Name 1|:Last name 2|:Phone 3|:Address 4
|| John .. | Smack | 111111111 |  Address1
|| Smith .. | Moon |222222222  |  Address2
|| Jenkins .. | Stone | 333333333 |  Address3
|| Ford .. | Mustang |444444444  |  Address4
								</content>
								<expected>Expected result for the third step</expected>
								<additional_info>Additional step 2 description (Var1: %Var1% and Var2: %Var2%)</additional_info>
							</step>
							<step>
								<index>5</index>
								<content> ![](index.php?/attachments/get/3b93e38b-5609-49c6-a987-8f2f6968121e) </content>
								<expected> ![](index.php?/attachments/get/6c9ccdd6-446f-4b23-9244-3a16b997a0e6) </expected>
								<additional_info> ![](index.php?/attachments/get/5edccf31-330d-4e62-9e8a-324f771d96c8) </additional_info>
							</step>
						</steps_separated>
					</custom>
				</case>
			</cases>
		</section>
	</sections>
			...


After running the script that converts this XML into a CSV file, we can see that , each Test is created and the steps are filled with the information extracted from the XML.

Code Block
languagetext
titleManual Tests in comic_estore.csv
collapsetrue
Issue ID,Issue Key,Test Type,Test Summary,Test Priority,Action,Data,Result,Test Repo,Labels
1,,Manual,Test Case Steps (Text),3,"%Var3 
%Var4
",, ![]([Link|https://testrail101.testrail.io/index.php?/attachments/get/d8d1d9e0-fe4c-43c8-b894-dfd83d9c160c)] ,Test Cases,Destructive
2,,Manual,Manual Test Case (Steps),3,"Login with user *%User1* 
Expected Result:
Application redirects you to the main page in a logged in state",,,Test Cases/Regression,UAT
2,,Manual,,3,First step description,Additional step description,Expected result for the first step,,
2,,Manual,,3,Second step description with two variables: %Var1% and %Var2% ,Additional step 2 description,Expected result for the second step,,
2,,Manual,,3,"Third step with a table:




|||:Name 1|:Last name 2|:Phone 3|:Address 4
|| John .. | Smack | 111111111 |  Address1
|| Smith .. | Moon |222222222  |  Address2
|| Jenkins .. | Stone | 333333333 |  Address3
|| Ford .. | Mustang |444444444  |  Address4
",Additional step 2 description (Var1: %Var1% and Var2: %Var2%),Expected result for the third step,,
2,,Manual,,3, ![]([Link|https://testrail101.testrail.io/index.php?/attachments/get/3b93e38b-5609-49c6-a987-8f2f6968121e)] , ![]([Link|https://testrail101.testrail.io/index.php?/attachments/get/5edccf31-330d-4e62-9e8a-324f771d96c8)] , ![]([Link|https://testrail101.testrail.io/index.php?/attachments/get/6c9ccdd6-446f-4b23-9244-3a16b997a0e6)] ,,


The first Test corresponds to a Manual Test Case with the summary imported obtained from the CSV file, one step with Action and Expected Result, and the added type (Destructive) as a Label.

Notice that we have added one image in the Expected Result in TestRail and the exported link is missing the TestRail endpoint. To obtain the full URL of the attachment you need to add your TestRail endpoint to the start of the link of the image, using the endpoint obtained from concatenating the TestRail endpoint and the attachment URL you will be able to access it.TestRail, and the exported link is reconstructed using the -e option of the script. 

Image AddedImage Removed


The second Test has one line per each step. Notice that the format is kept for the table and styles added in TestRail.

Image RemovedImage Added

Other Tests

The script will convert all tests of type type Manual, ; all the other types will be discarded, and no output is will be created in the resulting CSV file.

Info

The Type field (e.g., "Destructive", "Smoke", "UAT, etc") in TestRail is converted into labels and added in the Test issues managed by Xray.



Warning

This tutorial does not cover BDD Test Cases are not covered in this tutorial due to a current TestRail limitation in TestRail.



Tips

  • Use the CSV file provided to import the examples and the Json the Test Case Importer configuration file associated with it in order to have the setup and mapping correct.
  • Make sure Ensure you are using the scripts available for the right version of Jira (Cloud or Server).

Table of Contents
classtoc-btf

CSS Stylesheet
.toc-btf {
    position: fixed;
}