Page History
...
| Info |
|---|
| When manually importing execution results, the current Test Execution issue will be updated with the results. When using the REST API, you can specify an existing Test Execution issue in Jira; if . If the Test Execution key is missing, Xray will create a new Test Execution issue based on the information provided. |
...
The Cucumber tool is capable of generating multiple reports for an execution. In order to import the execution results to Xray, Cucumber must generate a JSON output (see example here) using the following arguments:
...
Xray JSON format
Xray also supports a both custom and generic format formats for importing execution results. This allows you to import any execution result as long as it is in JSON format. This way, means you can import any execution result to Jira, either from automated or manual Tests.
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
const XraySchema = {
id: "/XraySchema",
type: "object",
properties: {
testExecutionKey: {
type: "string"
},
info: {
$ref: "/TestExecutionInfo"
},
tests: {
type: "array",
items: {
$ref: "/XrayTest"
},
minItems: 1
}
},
additionalProperties: false
};
const TestExecutionInfo = {
id: "/TestExecutionInfo",
type: "object",
properties: {
project: {
type: "string"
},
summary: {
type: "string"
},
description: {
type: "string"
},
version: {
type: "string"
},
revision: {
type: "string"
},
user: {
type: "string"
},
startDate: {
type: "string",
format: "date-time"
},
finishDate: {
type: "string",
format: "date-time"
},
testPlanKey: {
type: "string"
},
testEnvironments: {
type: "array",
items: {
type: "string"
}
}
},
additionalProperties: false
};
const XrayTest = {
id: "/XrayTest",
oneOf: [
{
id: "XrayExistingTest",
type: "object",
properties: {
testKey: {
type: "string"
},
testInfoBean: {
$ref: "/TestInfo"
},
start: {
type: "string",
format: "date-time"
},
finish: {
type: "string",
format: "date-time"
},
comment: {
type: "string"
},
executedBy: {
type: "string"
},
assignee: {
type: "string"
},
status: {
type: "string"
},
results: {
type: "array",
items: {
$ref: "/ContextResult"
}
},
examples: {
type: "array",
items: {
type: "string"
}
},
steps: {
type: "array",
items: {
$ref: "/TestStepResult"
}
},
defects: {
type: "array",
items: {
type: "string"
}
},
evidences: {
type: "array",
items: {
$ref: "/Evidence"
}
}
},
required: ["status", "testKey"],
additionalProperties: false
},
{
id: "XrayNewTest",
type: "object",
properties: {
testInfoBean: {
oneOf: [
{
$ref: "/TestCreateInfoManual"
},
{
$ref: "/TestCreateInfoCucumber"
},
{
$ref: "/TestCreateInfoGeneric"
}
]
},
start: {
type: "string",
format: "date-time"
},
finish: {
type: "string",
format: "date-time"
},
comment: {
type: "string"
},
executedBy: {
type: "string"
},
assignee: {
type: "string"
},
status: {
type: "string"
},
results: {
type: "array",
items: {
$ref: "/ContextResult"
}
},
examples: {
type: "array",
items: {
type: "string",
enum: ["TODO", "FAILED", "PASSED", "EXECUTING"]
}
},
steps: {
type: "array",
items: {
$ref: "/TestStepResult"
}
},
defects: {
type: "array",
items: {
type: "string"
}
},
evidences: {
type: "array",
items: {
$ref: "/Evidence"
}
}
},
required: ["status", "testInfoBean"],
additionalProperties: false
}
]
};
const TestStepResult = {
id: "/TestStepResult",
type: "object",
properties: {
status: {
type: "string"
},
comment: {
type: "string"
},
evidences: {
type: "array",
items: {
$ref: "/Evidence"
}
},
defects: {
type: "array",
items: {
type: "string"
}
}
},
required: ["status"],
additionalProperties: false
};
const TestInfo = {
id: "/TestInfo",
type: "object",
properties: {
requirementKey: {
type: "string"
},
labels: {
type: "array",
items: {
type: "string"
}
}
},
additionalProperties: false
};
const TestCreateInfoManual = {
id: "/TestCreateInfoManual",
type: "object",
properties: {
summary: {
type: "string"
},
projectKey: {
type: "string"
},
manualTestDefinition: {
type: "array",
items: {
$ref: "/ManualStep"
}
},
requirementKey: {
type: "string"
},
labels: {
type: "array",
items: {
type: "string"
}
}
},
required: ["summary", "projectKey", "manualTestDefinition"],
additionalProperties: false
};
const TestCreateInfoCucumber = {
id: "/TestCreateInfoCucumber",
type: "object",
properties: {
summary: {
type: "string"
},
projectKey: {
type: "string"
},
cucumberTestDefinition: {
type: "string"
},
requirementKey: {
type: "string"
},
labels: {
type: "array",
items: {
type: "string"
}
}
},
required: ["summary", "projectKey", "cucumberTestDefinition"],
additionalProperties: false
};
const TestCreateInfoGeneric = {
id: "/TestCreateInfoGeneric",
type: "object",
properties: {
summary: {
type: "string"
},
projectKey: {
type: "string"
},
genericTestDefinition: {
type: "string"
},
requirementKey: {
type: "string"
},
labels: {
type: "array",
items: {
type: "string"
}
}
},
required: ["summary", "projectKey", "genericTestDefinition"],
additionalProperties: false
};
const ManualStep = {
id: "/ManualStep",
type: "object",
properties: {
action: {
type: "string"
},
data: {
type: "string"
},
result: {
type: "string"
}
},
required: ["action"],
additionalProperties: false
};
const Evidence = {
id: "/Evidence",
type: "object",
properties: {
data: {
type: "string"
},
filename: {
type: "string"
},
contentType: {
type: "string"
}
},
required: ["data", "filename"],
additionalProperties: false
};
const ContextResult = {
id: "/ContextResult",
type: "object",
properties: {
name: {
type: "string"
},
duration: {
type: "number"
},
log: { // not being used
type: "string"
},
status: {
type: "string",
enum: ["TODO", "FAILED", "PASSED", "EXECUTING"]
},
evidences: {
type: "array",
items: {
$ref: "/Evidence"
}
},
examples: {
type: "array",
items: {
type: "string",
enum: ["TODO", "FAILED", "PASSED", "EXECUTING"]
}
},
detailedExamples: {
type: "array",
items: {
$ref: "/Example"
}
}
},
required: ["name", "status"],
additionalProperties: false
};
const Example = {
id: "/Example",
type: "object",
properties: {
rank: {
type: "number"
},
duration: {
type: "number"
},
status: {
type: "string",
enum: ["TODO", "FAILED", "PASSED", "EXECUTING"]
},
steps: {
type: "array",
items: {
$ref: "/ExampleStep"
}
}
},
required: ["status"],
additionalProperties: false
};
const ExampleStep = {
id: "/ExampleStep",
type: "object",
properties: {
type: {
type: "string",
enum: ["hook", "background", "step"]
},
keyword: {
type: "string"
},
rank: {
type: "number"
},
name: {
type: "string"
},
duration: {
type: "number"
},
errorMessage: {
type: "string"
},
status: {
type: "string",
enum: ["TODO", "FAILED", "PASSED", "EXECUTING"]
},
evidences: {
type: "array",
items: {
$ref: "/Evidence"
}
}
},
required: ["type", "keyword", "name", "status"],
additionalProperties: false
}; |
Example 1:
...
Test with Test Type Gherkin or Unstructured
This is an example of a JSON file with execution results that will update Tests in Jira. The last item DEMO-9 must be a Test with a Test Type of kind Gherkin because the results contain examples.
...
| Code Block | ||
|---|---|---|
| ||
{
"tests" : [
{
"testKey" : "DEMO-7",
"start" : "2013-05-03T11:47:35+01:00",
"finish" : "2013-05-03T11:50:56+01:00",
"comment" : "Test was OK but the performance is very poor",
"status" : "PASS"
},
{
"testKey" : "DEMO-8",
"start" : "2013-05-03T12:14:12+01:00",
"finish" : "2013-05-03T12:15:23+01:00",
"comment" : "Performance was better this time, in the context of test set DEMO-10.",
"status" : "PASS"
},
{
"testKey" : "DEMO-9",
"start" : "2013-05-03T12:19:23+01:00",
"finish" : "2013-05-03T12:20:01+01:00",
"comment" : "Error decreasing space shuttle speed.",
"status" : "FAIL",
"examples" : [
"PASS",
"PASS",
"PASS",
"PASS",
"PASS",
"FAIL"
]
}
]
} |
Example 2: Tests
...
with Test Type Step
This is an example of a JSON file with execution results that will update a Test with a Test Type of kind Step in Jira.
| Code Block | ||
|---|---|---|
| ||
{
"tests" : [
{
"testKey" : "DEMO-57",
"start" : "2014-08-30T12:19:23+01:00",
"finish" : "2014-08-30T12:20:01+01:00",
"comment" : "Error executing step 2!",
"status" : "FAIL",
"steps": [
{
"status": "PASS",
"comment": "Step 1: OK"
},
{
"status": "FAIL",
"comment": "Step 2 *Failed* with an unexpected error message",
"evidences" : [
{
"data": "(... base 64 encoded ...)",
"filename": "screenshot1.jpg",
"contentType": "image/jpeg"
}
]
}
]
}
]
} |
...
Xray supports importing multiple results for the same Test issues issue in the same execution. These results often indicate different contexts/environments where the same Test must be was executed. Xray will group all executions of the same Test in a single Test Run and present all execution information, including the different contexts in the Execution page.
...
If you are using the Xamarin Test Cloud for executing Cucumber mobile Tests in different combinations of mobile devices and operating systems, you can import the results to Jira by making a compressed zip file containing the multiple Cucumber JSON files.
| Info | ||
|---|---|---|
| ||
Please see For more information, please refer to Testing using Calabash and Xamarin Test Cloud in Ruby and the "bundle" endpoint in the REST API. |