Page History
| Table of Contents | ||
|---|---|---|
|
External execution results from either automated and manual Tests can be imported to Jira. This operation may be done in one of two ways:
...
The Cucumber tool is capable of generating multiple reports for an execution. In order to import the execution results to Xray, Cucumber must generate a JSON output (example here) using the following arguments:
...
| Info | ||
|---|---|---|
| ||
You can import manual Test execution results using the steps element for specifying the Test step results. |
The JSON results file must follow this schema:
| Code Block | ||
|---|---|---|
| ||
{
"type": "object",
"$schema": "http://json-schema.org/draft-03/schema",
"required": false,
"properties": {
"testExecutionKey": {
"type": "string",
"required": false
},
"info": {
"type": "object",
"required": false,
"properties": {
"project": {
"type": "string",
"required": false
},
"summary": {
"type": "string",
"required": true
},
"description": {
"type": "string",
"required": false
},
"user": {
"type": "string",
"required": false
},
"version": {
"type": "string",
"required": false
},
"revision": {
"type": "string",
"required": false
},
"startDate": {
"type": "string",
"required": false
},
"finishDate": {
"type": "string",
"required": false
}
}
},
"tests": {
"type": "array",
"required": true,
"items": {
"type": "object",
"required": false,
"properties": {
"comment": {
"type": "string",
"required": false
},
"finish": {
"type": "string",
"required": false
},
"start": {
"type": "string",
"required": false
},
"status": {
"type": "string",
"required": true
},
"testKey": {
"type": "string",
"required": true
},
"executedBy": {
"type": "string",
"required": false
},
"evidences": {
"type": "array",
"required": false,
"items": {
"type": "object",
"required": false,
"properties": {
"data": {
"type": "string",
"required": true
},
"filename": {
"type": "string",
"required": true
},
"contentType": {
"type": "string",
"required": false
}
}
}
},
"results": {
"type": "array",
"required": false,
"items": {
"type": "object",
"required": false,
"properties": {
"name": {
"type": "string",
"required": true
},
"duration": {
"type": "long",
"required": false
},
"log": {
"type": "string",
"required": false
},
"status": {
"type": "string",
"required": true
},
"examples": {
"type": "array",
"required": false,
"items": {
"type": "string"
}
}
}
}
},
"examples": {
"type": "array",
"required": false,
"items": {
"type": "string"
}
},
"steps": {
"type": "array",
"required": false,
"items": {
"type": "object",
"required": false,
"properties": {
"status": {
"type": "string",
"required": true
},
"comment": {
"type": "string",
"required": false
},
"evidences": {
"type": "array",
"required": false,
"items": {
"type": "object",
"required": false,
"properties": {
"data": {
"type": "string",
"required": true
},
"filename": {
"type": "string",
"required": true
},
"contentType": {
"type": "string",
"required": false
}
}
}
}
}
}
}
}
}
}
}
} |
...
Example 1: Automated Tests
This is an example of a JSON file with execution results that will update Automated Tests in Jira. The last item DEMO-9 must be a Cucumber Test because the results contain examples.
| Code Block | ||
|---|---|---|
| ||
{
"tests" : [
{
"testKey" : "DEMO-7",
"start" : "2013-05-03T11:47:35+01:00",
"finish" : "2013-05-03T11:50:56+01:00",
"comment" : "Test was OK but the performance is very poor",
"status" : "PASS"
},
{
"testKey" : "DEMO-8",
"start" : "2013-05-03T12:14:12+01:00",
"finish" : "2013-05-03T12:15:23+01:00",
"comment" : "Performance was better this time, in the context of test set DEMO-10.",
"status" : "PASS"
},
{
"testKey" : "DEMO-9",
"start" : "2013-05-03T12:19:23+01:00",
"finish" : "2013-05-03T12:20:01+01:00",
"comment" : "Error decreasing space shuttle speed.",
"status" : "FAIL",
"examples" : [
"PASS",
"PASS",
"PASS",
"PASS",
"PASS",
"FAIL"
]
}
]
} |
...
Example 2: Manual Tests
This is an example of a JSON file with execution results that will update a Manual Test in Jira.
| Code Block | ||
|---|---|---|
| ||
{
"tests" : [
{
"testKey" : "DEMO-57",
"start" : "2014-08-30T12:19:23+01:00",
"finish" : "2014-08-30T12:20:01+01:00",
"comment" : "Error executing step 2!",
"status" : "FAIL",
"steps": [
{
"status": "PASS",
"comment": "Step 1: OK"
},
{
"status": "FAIL",
"comment": "Step 2 *Failed* with an unexpected error message",
"evidences" : [
{
"data": "(... base 64 encoded ...)",
"filename": "screenshot1.jpg",
"contentType": "image/jpeg"
}
]
}
]
}
]
} |
Example 3: Generic Test
This is an example of a JSON file with execution results that will update a Generic Test in Jira.
| Code Block | ||
|---|---|---|
| ||
{
"tests" : [
{
"testKey" : "ABC-129",
"start" : "2014-08-30T11:47:35+01:00",
"finish" : "2014-08-30T11:50:56+01:00",
"comment" : "Successful execution",
"status" : "PASS"
}
]
} |
...
Xray supports importing multiple results for the same Test issues in the same execution. These results often indicate different contexts/environments where the same Test must be executed. Xray will group all executions of the same Test in a single Test Run and present all execution information, including the different contexts in the Execution page.
| Tip |
|---|
When importing execution results using the Cucumber JSON output formatter, multiple executions for the same scenario/Test issue will be merged into a single Test Run. When importing execution results using the Xray JSON format, the results JSON element must contain all execution results for the same Test Run. |
...
If you are using the Xamarin Test Cloud for executing Cucumber mobile Tests in different combinations of mobile devices and operating systems, you can import the results to Jira by making a compressed zip file containing the multiple Cucumber JSON files.
| Info | ||
|---|---|---|
| ||
Please see Testing using Calabash and Xamarin Test Cloud in Ruby and the "bundle" endpoint in the REST API. |