Page History
...
External execution results from either automated and manual Tests can be imported to JIRAJira. This operation may be done in one of two ways:
- Manually, using the "Import Execution Results" action accessible from the Test Execution issue screen
- using via the REST API to integrate with Continuous Integration (CI) platforms or other external execution processes. Please check refer to the Xray REST API.
| Info |
|---|
| When manually importing execution results, using the action above, the current Test Execution issue will be updated with the results. Using When using the REST API, you can specify an existing Test Execution issue in JIRA or, in case Jira; if the Test Execution key is missing, Xray will create a new Test Execution issue based on the information provided information. |
Supported Import Formats
Xray for JIRA supports two distinct JSON supports the following formats for importing execution results:
...
The Cucumber tool is capable of generating multiple reports for an execution. In order to import the execution results to Xray, Cucumber must generate a JSON output (example here) using the following arguments:
...
NUnit v3.0 XML output format
Xray supports a an NUnit v3.0 XML format for importing execution results.
...
NUnit v2.6 XML output format
Xray supports a an NUnit v2.6 XML format for importing execution results.
...
Xray also supports a custom and generic format for importing execution results. This allows you to import any execution result as long create a JSON results file. Using this formatas it is in JSON format. This way, you can import any execution result to JIRAJira, either from automated or manual Tests.
...
| Info | ||
|---|---|---|
| ||
The "info" object allows you to specify specific execution information when creating new Test Execution issues. |
...
| Info | ||
|---|---|---|
| ||
You can import manual Test execution results using the "steps " element for specifying the Test step results. |
...
| Code Block | ||
|---|---|---|
| ||
{
"type": "object",
"$schema": "http://json-schema.org/draft-03/schema",
"required": false,
"properties": {
"testExecutionKey": {
"type": "string",
"required": false
},
"info": {
"type": "object",
"required": false,
"properties": {
"project": {
"type": "string",
"required": false
},
"summary": {
"type": "string",
"required": true
},
"description": {
"type": "string",
"required": false
},
"user": {
"type": "string",
"required": false
},
"version": {
"type": "string",
"required": false
},
"revision": {
"type": "string",
"required": false
},
"startDate": {
"type": "string",
"required": false
},
"finishDate": {
"type": "string",
"required": false
}
}
},
"tests": {
"type": "array",
"required": true,
"items": {
"type": "object",
"required": false,
"properties": {
"comment": {
"type": "string",
"required": false
},
"finish": {
"type": "string",
"required": false
},
"start": {
"type": "string",
"required": false
},
"status": {
"type": "string",
"required": true
},
"testKey": {
"type": "string",
"required": true
},
"executedBy": {
"type": "string",
"required": false
},
"evidences": {
"type": "array",
"required": false,
"items": {
"type": "object",
"required": false,
"properties": {
"data": {
"type": "string",
"required": true
},
"filename": {
"type": "string",
"required": true
},
"contentType": {
"type": "string",
"required": false
}
}
}
},
"results": {
"type": "array",
"required": false,
"items": {
"type": "object",
"required": false,
"properties": {
"name": {
"type": "string",
"required": true
},
"duration": {
"type": "long",
"required": false
},
"log": {
"type": "string",
"required": false
},
"status": {
"type": "string",
"required": true
},
"examples": {
"type": "array",
"required": false,
"items": {
"type": "string"
}
}
}
}
},
"examples": {
"type": "array",
"required": false,
"items": {
"type": "string"
}
},
"steps": {
"type": "array",
"required": false,
"items": {
"type": "object",
"required": false,
"properties": {
"status": {
"type": "string",
"required": true
},
"comment": {
"type": "string",
"required": false
},
"evidences": {
"type": "array",
"required": false,
"items": {
"type": "object",
"required": false,
"properties": {
"data": {
"type": "string",
"required": true
},
"filename": {
"type": "string",
"required": true
},
"contentType": {
"type": "string",
"required": false
}
}
}
}
}
}
}
}
}
}
}
} |
Example 1
...
: Automated Tests
This is an example of an a JSON file with execution results that will update Automated Tests in JIRAJira. The last example item DEMO-9 must be a Cucumber Test because the results contain examples.
...
| Code Block | ||
|---|---|---|
| ||
{
"tests" : [
{
"testKey" : "DEMO-7",
"start" : "2013-05-03T11:47:35+01:00",
"finish" : "2013-05-03T11:50:56+01:00",
"comment" : "Test was OK but the performance is very poor",
"status" : "PASS"
},
{
"testKey" : "DEMO-8",
"start" : "2013-05-03T12:14:12+01:00",
"finish" : "2013-05-03T12:15:23+01:00",
"comment" : "Performance was better this time, in the context of test set DEMO-10.",
"status" : "PASS"
},
{
"testKey" : "DEMO-9",
"start" : "2013-05-03T12:19:23+01:00",
"finish" : "2013-05-03T12:20:01+01:00",
"comment" : "Error decreasing space shuttle speed.",
"status" : "FAIL",
"examples" : [
"PASS",
"PASS",
"PASS",
"PASS",
"PASS",
"FAIL"
]
}
]
} |
Example 2
...
: Manual Tests
This is an example of an a JSON file with execution results that will update a Manual Test in JIRAJira.
| Code Block | ||
|---|---|---|
| ||
{
"tests" : [
{
"testKey" : "DEMO-57",
"start" : "2014-08-30T12:19:23+01:00",
"finish" : "2014-08-30T12:20:01+01:00",
"comment" : "Error executing step 2!",
"status" : "FAIL",
"steps": [
{
"status": "PASS",
"comment": "Step 1: OK"
},
{
"status": "FAIL",
"comment": "Step 2 *Failed* with an unexpected error message",
"evidences" : [
{
"data": "(... base 64 encoded ...)",
"filename": "screenshot1.jpg",
"contentType": "image/jpeg"
}
]
}
]
}
]
} |
Example 3
...
: Generic
...
Test
This is an example of an a JSON file with execution results that will update a Generic Test in JIRAJira.
| Code Block | ||
|---|---|---|
| ||
{
"tests" : [
{
"testKey" : "ABC-129",
"start" : "2014-08-30T11:47:35+01:00",
"finish" : "2014-08-30T11:50:56+01:00",
"comment" : "Successful execution",
"status" : "PASS"
}
]
} |
...
Xray supports importing multiple results for the same Test issues in the same execution. These results often indicate different contextcontexts/environments where the same Test must be executed. Xray will group all executions of the same Test in a single Test Run and present all execution information, including it's the different contexts in the Execution page.
...
| Tip |
|---|
When importing execution results using the Cucumber JSON output formatter, then multiple executions for the same scenario/Test issue will be merged into a single Test Run. When importing execution results with results using the Xray JSON format, the "results" JSON element must contain all execution results for the same Test Run. |
Xamarin Test Cloud
If you are using the Xamarin Test Cloud for executing Cucumber mobile Tests in different combinations of mobile devices and operating systems, you can import the results to JIRA Jira by making a compressed zip file containing the multiple Cucumber JSON files.
...