External execution results from either automated and manual Tests can be imported to Jira. This operation may be done in one of two ways:
- Manually, using the Import Execution Results action accessible from the Test Execution issue screen
- via the REST API to integrate with Continuous Integration (CI) platforms or other external execution processes. Please refer to the Xray REST API.
Supported Import Formats
Xray supports the following formats for importing execution results:
- Cucumber JSON output format
- Xray JSON format
Cucumber JSON output format
The Cucumber tool is capable of generating multiple reports for an execution. In order to import the execution results to Xray, Cucumber must generate a JSON output (see example here) using the following arguments:
-f, --format FORMAT How to format features. Available formats compatible with Xray for JIRA:
json : Prints the feature as JSON
json_pretty : Prints the feature as prettified JSON
-x, --expand Expand Scenario Outline Tables in output.
Ex:
> cucumber -x -f json
Xray JSON format
Xray also supports both custom and generic formats for importing execution results. This allows you to import any execution result as long as it is in JSON format. This means you can import any execution result to Jira, either from automated or manual Tests.
Importing Execution Evidences
Creating new Test Execution issues
The info object allows you to define specific execution information when creating new Test Execution issues.
Importing results to Manual Tests
You can import manual Test execution results using the steps element for specifying the Test step results.
The JSON results file must follow this schema:
const XraySchema = {
id: "/XraySchema",
type: "object",
properties: {
testExecutionKey: {
type: "string"
},
info: {
$ref: "/TestExecutionInfo"
},
tests: {
type: "array",
items: {
$ref: "/XrayTest"
},
minItems: 1
}
},
additionalProperties: false
};
const TestExecutionInfo = {
id: "/TestExecutionInfo",
type: "object",
properties: {
project: {
type: "string"
},
summary: {
type: "string"
},
description: {
type: "string"
},
version: {
type: "string"
},
revision: {
type: "string"
},
user: {
type: "string"
},
startDate: {
type: "string",
format: "date-time"
},
finishDate: {
type: "string",
format: "date-time"
},
testPlanKey: {
type: "string"
},
testEnvironments: {
type: "array",
items: {
type: "string"
}
}
},
additionalProperties: false
};
const XrayTest = {
id: "/XrayTest",
oneOf: [
{
id: "XrayExistingTest",
type: "object",
properties: {
testKey: {
type: "string"
},
testInfoBean: {
$ref: "/TestInfo"
},
start: {
type: "string",
format: "date-time"
},
finish: {
type: "string",
format: "date-time"
},
comment: {
type: "string"
},
executedBy: {
type: "string"
},
assignee: {
type: "string"
},
status: {
type: "string"
},
results: {
type: "array",
items: {
$ref: "/ContextResult"
}
},
examples: {
type: "array",
items: {
type: "string"
}
},
steps: {
type: "array",
items: {
$ref: "/TestStepResult"
}
},
defects: {
type: "array",
items: {
type: "string"
}
},
evidences: {
type: "array",
items: {
$ref: "/Evidence"
}
}
},
required: ["status", "testKey"],
additionalProperties: false
},
{
id: "XrayNewTest",
type: "object",
properties: {
testInfoBean: {
oneOf: [
{
$ref: "/TestCreateInfoManual"
},
{
$ref: "/TestCreateInfoCucumber"
},
{
$ref: "/TestCreateInfoGeneric"
}
]
},
start: {
type: "string",
format: "date-time"
},
finish: {
type: "string",
format: "date-time"
},
comment: {
type: "string"
},
executedBy: {
type: "string"
},
assignee: {
type: "string"
},
status: {
type: "string"
},
results: {
type: "array",
items: {
$ref: "/ContextResult"
}
},
examples: {
type: "array",
items: {
type: "string",
enum: ["TODO", "FAILED", "PASSED", "EXECUTING"]
}
},
steps: {
type: "array",
items: {
$ref: "/TestStepResult"
}
},
defects: {
type: "array",
items: {
type: "string"
}
},
evidences: {
type: "array",
items: {
$ref: "/Evidence"
}
}
},
required: ["status", "testInfoBean"],
additionalProperties: false
}
]
};
const TestStepResult = {
id: "/TestStepResult",
type: "object",
properties: {
status: {
type: "string"
},
comment: {
type: "string"
},
evidences: {
type: "array",
items: {
$ref: "/Evidence"
}
},
defects: {
type: "array",
items: {
type: "string"
}
}
},
required: ["status"],
additionalProperties: false
};
const TestInfo = {
id: "/TestInfo",
type: "object",
properties: {
requirementKey: {
type: "string"
},
labels: {
type: "array",
items: {
type: "string"
}
}
},
additionalProperties: false
};
const TestCreateInfoManual = {
id: "/TestCreateInfoManual",
type: "object",
properties: {
summary: {
type: "string"
},
projectKey: {
type: "string"
},
manualTestDefinition: {
type: "array",
items: {
$ref: "/ManualStep"
}
},
requirementKey: {
type: "string"
},
labels: {
type: "array",
items: {
type: "string"
}
}
},
required: ["summary", "projectKey", "manualTestDefinition"],
additionalProperties: false
};
const TestCreateInfoCucumber = {
id: "/TestCreateInfoCucumber",
type: "object",
properties: {
summary: {
type: "string"
},
projectKey: {
type: "string"
},
cucumberTestDefinition: {
type: "string"
},
requirementKey: {
type: "string"
},
labels: {
type: "array",
items: {
type: "string"
}
}
},
required: ["summary", "projectKey", "cucumberTestDefinition"],
additionalProperties: false
};
const TestCreateInfoGeneric = {
id: "/TestCreateInfoGeneric",
type: "object",
properties: {
summary: {
type: "string"
},
projectKey: {
type: "string"
},
genericTestDefinition: {
type: "string"
},
requirementKey: {
type: "string"
},
labels: {
type: "array",
items: {
type: "string"
}
}
},
required: ["summary", "projectKey", "genericTestDefinition"],
additionalProperties: false
};
const ManualStep = {
id: "/ManualStep",
type: "object",
properties: {
action: {
type: "string"
},
data: {
type: "string"
},
result: {
type: "string"
}
},
required: ["action"],
additionalProperties: false
};
const Evidence = {
id: "/Evidence",
type: "object",
properties: {
data: {
type: "string"
},
filename: {
type: "string"
},
contentType: {
type: "string"
}
},
required: ["data", "filename"],
additionalProperties: false
};
const ContextResult = {
id: "/ContextResult",
type: "object",
properties: {
name: {
type: "string"
},
duration: {
type: "number"
},
log: { // not being used
type: "string"
},
status: {
type: "string",
enum: ["TODO", "FAILED", "PASSED", "EXECUTING"]
},
evidences: {
type: "array",
items: {
$ref: "/Evidence"
}
},
examples: {
type: "array",
items: {
type: "string",
enum: ["TODO", "FAILED", "PASSED", "EXECUTING"]
}
},
detailedExamples: {
type: "array",
items: {
$ref: "/Example"
}
}
},
required: ["name", "status"],
additionalProperties: false
};
const Example = {
id: "/Example",
type: "object",
properties: {
rank: {
type: "number"
},
duration: {
type: "number"
},
status: {
type: "string",
enum: ["TODO", "FAILED", "PASSED", "EXECUTING"]
},
steps: {
type: "array",
items: {
$ref: "/ExampleStep"
}
}
},
required: ["status"],
additionalProperties: false
};
const ExampleStep = {
id: "/ExampleStep",
type: "object",
properties: {
type: {
type: "string",
enum: ["hook", "background", "step"]
},
keyword: {
type: "string"
},
rank: {
type: "number"
},
name: {
type: "string"
},
duration: {
type: "number"
},
errorMessage: {
type: "string"
},
status: {
type: "string",
enum: ["TODO", "FAILED", "PASSED", "EXECUTING"]
},
evidences: {
type: "array",
items: {
$ref: "/Evidence"
}
}
},
required: ["type", "keyword", "name", "status"],
additionalProperties: false
};
Example 1: Test with Test Type Gherkin or Unstructured
This is an example of a JSON file with execution results that will update Tests in Jira. The last item DEMO-9 must be a Test with a Test Type of Gherkin because the results contain examples.
{
"tests" : [
{
"testKey" : "DEMO-7",
"start" : "2013-05-03T11:47:35+01:00",
"finish" : "2013-05-03T11:50:56+01:00",
"comment" : "Test was OK but the performance is very poor",
"status" : "PASSED"
},
{
"testKey" : "DEMO-8",
"start" : "2013-05-03T12:14:12+01:00",
"finish" : "2013-05-03T12:15:23+01:00",
"comment" : "Performance was better this time, in the context of test set DEMO-10.",
"status" : "PASSED"
},
{
"testKey" : "DEMO-9",
"start" : "2013-05-03T12:19:23+01:00",
"finish" : "2013-05-03T12:20:01+01:00",
"comment" : "Error decreasing space shuttle speed.",
"status" : "FAILED",
"examples" : [
"PASSED",
"PASSED",
"PASSED",
"PASSED",
"PASSED",
"FAILED"
]
}
]
}
Example 2: Tests with Test Type Step
This is an example of a JSON file with execution results that will update a Test with a Test Type of Step in Jira.
{
"tests" : [
{
"testKey" : "DEMO-57",
"start" : "2014-08-30T12:19:23+01:00",
"finish" : "2014-08-30T12:20:01+01:00",
"comment" : "Error executing step 2!",
"status" : "FAILED",
"steps": [
{
"status": "PASSED",
"comment": "Step 1: OK"
},
{
"status": "FAILED",
"comment": "Step 2 *Failed* with an unexpected error message",
"evidences" : [
{
"data": "(... base 64 encoded ...)",
"filename": "screenshot1.jpg",
"contentType": "image/jpeg"
}
]
}
]
}
]
}
Importing Multiple Execution results
Xray supports importing multiple results for the same Test issue in the same execution. These results often indicate different contexts/environments where the same Test was executed. Xray will group all executions of the same Test in a single Test Run and present all execution information, including the different contexts in the Execution page.
When importing execution results using the Cucumber JSON output formatter, multiple executions for the same scenario/Test issue will be merged into a single Test Run.
When importing execution results using the Xray JSON format, the results JSON element must contain all execution results for the same Test Run.
Xamarin Test Cloud
If you are using the Xamarin Test Cloud for executing Cucumber mobile Tests in different combinations of mobile devices and operating systems, you can import the results to Jira by making a compressed zip file containing the multiple Cucumber JSON files.
Learn more
For more information, please refer to Testing using Calabash and Xamarin Test Cloud in Ruby and the "bundle" endpoint in the REST API.