Page History
Table of Contents | ||
---|---|---|
|
External execution results from either automated and manual Tests can be imported to Jira. This operation may be done in one of two ways:
...
Overview
Xray provides a proprietary JSON format for importing execution results into Jira/Xray.
Although Xray supports multiple report formats used by different testing frameworks/runners (e.g. JUnit, NUnit, xUnit, TestNG, Cucumber, Robot Framework), there are scenarios where using these formats is not an option like:
- Using a testing framework report that is not supported by Xray
- Having your own testing framework
- Limited support of existing formats to import detailed execution results back into Jira
JSON format
testExecutionKey | The test execution key where to import the execution results |
info | The info object for creating new Test Execution issues (link) |
tests | The Test Run result details (link) |
Anchor | ||||
---|---|---|---|---|
|
"info" object - Test Execution issue
You can specify which Test Execution issue to import the results by setting the test execution key on the testExecutionKey property. Alternatively, you can create a new Test Execution issue for the execution results and specify the issue fields within the info object.
project | The project key where the test execution will be created |
summary | The summary for the test execution issue |
description | The description for the test execution issue |
version | The version name for the Fix Version field of the test execution issue |
revision | A revision for the revision custom field |
user | The userid for the Jira user who executed the tests |
startDate | The start date for the test execution issue |
finishDate | The finish date for the test execution issue |
testPlanKey | The test plan key for associating the test execution issue |
testEnvironments | The test environments for the test execution issue |
Anchor | ||||
---|---|---|---|---|
|
"test" object - Test Run details
The test run details object allows you to import any detail about the execution itself. All Xray test types are supported.
It is possible to import a single result (the test object itself with the "steps" (Manual tests) or "examples" (BDD tests)) or multiple execution results into the same Test Run (data-driven testing) using the "iterations" array.
testKey | The test issue key |
testInfo | The testInfo element (link) |
start | The start date for the test run |
finish | The finish date for the test run |
comment | The comment for the test run |
executedBy | The user id who executed the test run |
assignee | The user id for the assignee of the test run |
status | The test run status (PASSED, FAILED, EXECUTING, TODO, custom statuses ...) |
steps | The step results (link) |
examples | The example results for BDD tests (link) |
iterations | The iteration containing data-driven test results (link) |
defects | An array of defect issue keys to associate with the test run |
evidence | An array of evidence items of the test run (link) |
customFields | An array of custom fields for the test run (link) |
Anchor testinfo testinfo
"testInfo" object - Creating Test issues
It is possible to create new test issues when importing execution results using the Xray JSON format. For this, a testInfo element must be provided in order for Xray to create the issues.
If it is the first time you are importing an execution with a testInfo, Xray will create the tests automatically. Sub-sequent executions will reuse the same test issues.
Info |
---|
Xray will first try to match test issues by the testKey if present. Otherwise, Manual or BDD tests are matched by summary whilst Generic tests are matched using the generic definition field, within the same project. |
Any changes to the testInfo element will update the test issue specification in Jira.
Note |
---|
If the match field (summary or definition) is changed, Xray will search for another issue and will create a new test case, or update an existing test case if no one is found. If you need to change the summary or the definition, you can do it manually (go to Jira and change the field), or you can include the testKey within the test element. |
projectKey | The project key where the test issue will be created |
summary | The summary for the test issue |
type | The test type (e.g. Manual, Cucumber, Generic) |
requirementKeys | An array of requirement issue keys to associate with the test |
labels | The test issue labels |
steps | An array of test steps (for Manual tests) (link) |
scenario | The BDD scenario |
definition | The generic test definition |
Anchor | ||||
---|---|---|---|---|
|
"step def" object - step definition
This object allows you to define the step fields for manual tests. Custom test step fields are also supported.
action | The step action - native field |
data | The step data - native field |
result | The step expected result - native field |
(...) | Any other step custom fields |
Anchor iteration iteration
"iteration" object - Data-driven test results
If you need to import data-driven test results you need to use the iteration object. Xray will store all iterations within the same Test Run object.
It is also possible to import iteration results with parameters. Currently, this is only supported for manual tests.
In this case, Xray will create a dataset automatically within the Test Run object.
name | The iteration name |
parameters | An array of parameters along with their values (link) |
log | The log for the iteration |
duration | A duration for the iteration |
status | The status for the iteration (PASSED, FAILED, EXECUTING, TODO, custom statuses ...) |
steps | An array of step results (for Manual tests) (link) |
Anchor | ||||
---|---|---|---|---|
|
"parameter" object - parameters within iteration results
name | The parameter name |
value | The parameter value |
Anchor | ||||
---|---|---|---|---|
|
"step result" object - step results
status | The status for the test step (PASSED, FAILED, EXECUTING, TODO, custom statuses ...) |
comment | The comment for the step result |
actualResult | The actual result field for the step result |
evidence | An array of evidence items of the test run (link) |
defects | An array of defect issue keys to associate with the test run |
Anchor | ||||
---|---|---|---|---|
|
"evidence" object - embedded attachments
data | The attachment data encoded in base64 |
filename | The file name for the attachment |
contentType | The Content-Type representation header is used to indicate the original media type of the resource |
Anchor | ||||
---|---|---|---|---|
|
"customField" object - store test run custom fields
It is possible to import test run custom field values into the Test Run object. Xray will use the "id" or "name" to find the existing test run custom field in the project settings.
id | The test run custom field ID |
name | The test run custom field name |
value | The test run custom field value |
Xray JSON Schema
The JSON results file must comply to the following JSON Schema:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
{
"$id": "XraySchema",
"type": "object",
"properties": {
"testExecutionKey": {
"type": "string"
},
"info": {
"type": "object",
"properties": {
"project": {
"type": "string"
},
"summary": {
"type": "string"
},
"description": {
"type": "string"
},
"version": {
"type": "string"
},
"revision": {
"type": "string"
},
"user": {
"type": "string"
},
"startDate": {
"type": "string",
"format": "date-time"
},
"finishDate": {
"type": "string",
"format": "date-time"
},
"testPlanKey": {
"type": "string"
},
"testEnvironments": {
"type": "array",
"items" |
...
Info |
---|
When manually importing execution results, the current Test Execution issue will be updated with the results. When using the REST API, you can specify an existing Test Execution issue in Jira. If the Test Execution key is missing, Xray will create a new Test Execution issue based on the information provided. |
Supported Import Formats
Xray supports the following formats for importing execution results:
- Cucumber JSON output format
- Xray JSON format
Cucumber JSON output format
The Cucumber tool is capable of generating multiple reports for an execution. In order to import the execution results to Xray, Cucumber must generate a JSON output (see example here) using the following arguments:
Code Block | ||
---|---|---|
| ||
-f, --format FORMAT How to format features. Available formats compatible with Xray for JIRA:
json : Prints the feature as JSON
json_pretty : Prints the feature as prettified JSON
-x, --expand Expand Scenario Outline Tables in output.
Ex:
> cucumber -x -f json |
Xray JSON format
Xray also supports both custom and generic formats for importing execution results. This allows you to import any execution result as long as it is in JSON format. This means you can import any execution result to Jira, either from automated or manual Tests.
Info | ||
---|---|---|
| ||
This format supports importing execution evidences (attachments) embedded in the JSON file encoded in Base 64. |
Info | ||
---|---|---|
| ||
The info object allows you to define specific execution information when creating new Test Execution issues. |
Info | ||
---|---|---|
| ||
You can import manual Test execution results using the steps element for specifying the Test step results. |
The JSON results file must follow this schema:
Code Block | ||||
---|---|---|---|---|
| ||||
const XraySchema = { id: "/XraySchema", type: "object", properties: { testExecutionKey: { type: "string" }, info: { $ref: "/TestExecutionInfo" }, tests: { type: "array", items: { $ref: "/XrayTest" }, minItems: 1 } }, additionalProperties: false }; const TestExecutionInfo = { id: "/TestExecutionInfo", type: "object", properties: { project: { "type": "string" } } }, "additionalProperties": false }, summary"tests": { type"type": "stringarray", "items": { }, description"$ref": {"#/definitions/Test" }, type"minItems": "string"1, } }, "additionalProperties": false, version"definitions": { "Test": { "type": "stringobject", },"properties": { revision"testKey": { "type": "string" }, user"testInfo": { type"$ref": "string#/definitions/TestInfo" }, startDate"start": { "type": "string", "format": "date-time" }, finishDate"finish": { "type": "string", "format": "date-time" }, testPlanKey"comment": { "type": "string" }, testEnvironments"executedBy": { "type": "arraystring", }, items: { "assignee": { "type": "string" } } }, additionalProperties: false };, const XrayTest = { id: "/XrayTest",status": { oneOf: [ "type": "string" { }, id: "XrayExistingTeststeps",: { "type": "objectarray", properties"items": { testKey: {"$ref": "#/definitions/ManualTestStepResult" } type: "string"}, "examples": { }, "type": "array", testInfoBean"items": { $ref"type": "/TestInfostring", "enum": ["TODO", "FAILED", },"PASSED", "EXECUTING"] } start: {}, "results": { // DEPRECATED "type": "stringarray", format: "date-time""items": { },"$ref": "#/definitions/IterationResult" } finish: {}, "iterations": { "type": "stringarray", "items": { format"$ref": "date-time#/definitions/IterationResult" } }, "defects": { comment "type": {"array", "items": { "type": "string" } }, "evidence": { "type": "array", executedBy"items": { type"$ref": "string#/definitions/EvidenceItem" }, }, assignee"evidences": { // DEPRECATED "type": "array", type: "stringitems": { "$ref": "#/definitions/EvidenceItem" }, } status: {}, "customFields": { type"$ref": "string#/definitions/CustomField" } }, "required": ["status"], results"dependencies": { "evidence": { "not": { type"required": ["arrayevidences",] } }, items"evidences": { "not": { "required": ["evidence"] } $ref: "/ContextResult"}, "steps": { "allOf": }[ }, { examples"not": { "required": ["examples"] } }, type: "array", { items"not": { "required": ["results"] } }, type: "string" { "not": { "required": ["iterations"] } } }, ] }, steps: { "examples": { "allOf": [ type: "array", { items "not": { "required": ["steps"] } }, $ref: "/TestStepResult" { "not": { "required": ["results"] } }, defects: { "not": type{ "required": ["arrayiterations",] } } items: {] }, "results": { type: "stringallOf": [ { } "not": { "required": ["steps"] } }, }, evidences: { "not": type{ "required": ["arrayexamples",] } }, items: { "not": { "required": ["iterations"] } $ref: "/Evidence" } ] }, "iterations": { "allOf": }[ },{ required: ["status", "testKey"], "not": { "required": ["steps"] } additionalProperties: false}, }, { { "not": { id"required": ["XrayNewTestexamples",] } type: "object"}, properties: { "not": { testInfoBean"required": { ["results"] } } oneOf: [] } }, "additionalProperties": false }, "IterationResult": { "type": "object", "properties": { "name": { $ref: "/TestCreateInfoManual" "type": "string" }, },"parameters": { "type": "array", "items": { "type": "object", $ref"properties": "/TestCreateInfoCucumber"{ "name": { }, "type": "string" {}, "value": { $ref "type": "/TestCreateInfoGenericstring" } }, "required": ["name"], ] "additionalProperties": false }, }, start"log": { "type": "string" type: "string"}, "duration": { format"type": "date-timestring" }, "status": { }, "type": "string" finish: {}, "steps": { "type": "stringarray", "items": { format"$ref": "date-time#/definitions/ManualTestStepResult" } }, }, "required": ["status"], comment: { "additionalProperties": false }, "ManualTestStepResult": { "type": "stringobject", "properties": { "status": { }, "type": "string" executedBy: {}, "comment": { "type": "string" }, }, "evidence": { "type": "array", assignee: { "items": { type"$ref": "string#/definitions/EvidenceItem" } }, "defects": { status "type": { "array", "items": { "type": "string" } }, "actualResult": { results"type": {"string" } }, type"required": ["arraystatus"], "additionalProperties": false }, "TestInfo": { items "type": {"object", "properties": { "summary": { $ref "type": "/ContextResultstring" }, } "projectKey": { },"type": "string" }, examples"requirementKeys": { "type": "array", type: "arrayitems",: { "type": "string" items: {} }, "labels": { "type": "stringarray", "items": { enum: ["TODOtype",: "FAILEDstring", "PASSED", "EXECUTING"] } }, } "type": { },"type": "string" }, steps: { type: "array", items: { items type: "object", properties: { action: { $reftype: "/TestStepResultstring" }, data: },{ defectstype: {"string" }, type: "array", result: { items: { type: "string" } type: "string" }, // custom }fields patternProperties: { }, ".+": {} evidences: { }, typerequired: ["arrayaction"], additionalProperties: false items: {} }, "scenario": { $ref"type": "/Evidencestring" }, "definition": { } "type": "string" } }, "dependencies": { }, "steps": { required: ["statusallOf", "testInfoBean"],: [ additionalProperties: false{ } ] }; const TestStepResult ="not": { id "required": ["/TestStepResult", scenario"] } type: "object", properties: {}, status: { type: "stringnot" : { "required": ["definition"] }, comment: { } type: "string"] }, evidences"scenario": { type"allOf": "array",[ items: { "not": { $ref"required": ["/Evidence"steps"] } }, }, { defects: { "not": { type"required": ["arraydefinition",] } items: {} ] }, type"definition": "string"{ }"allOf": [ } },{ required: ["status"], additionalProperties: false }; const TestInfo = { id: "/TestInfonot", : type{ "required": ["objectsteps", ] } properties: { requirementKey: {}, type: "string"{ }, "not": { labels"required": { ["scenario"] } type: "array", } ] items: { } }, type: "string""required": ["summary", "projectKey", "type"], "additionalProperties": false }, "EvidenceItem": { } }"type": "object", additionalProperties"properties": false{ }; const TestCreateInfoManual = { id: "/TestCreateInfoManual", data": { "type": "objectstring", properties: {}, summary"filename": { "type": "string" }, projectKey"contentType": { "type": "string" }, manualTestDefinition: { }, "required": ["data", "filename"], type: "arrayadditionalProperties",: false }, items"CustomField": { "type": "array", $ref"items": "/ManualStep"{ "type": "object", } "properties": { }, requirementKey"id": { "type": "string" }, labels"name": { "type": "arraystring", }, items "value": {} }, type"anyOf": "string"[ }{ } }, required"required": ["summaryid", "projectKey", "manualTestDefinition"],value"] additionalProperties: false }; const TestCreateInfoCucumber = { }, { id"required": ["/TestCreateInfoCucumbername", "value"] type: "object", properties: {} summary: {], "additionalProperties": false type: "string" } } } }, projectKey: { type: "string" }, |
Examples
Importing gherkin and other test results
In this example, we are importing execution results for three existing test issues in Jira. The last issue DEMO-9 must be a BDD Test with a Gherkin definition because the results contain examples. The remaining issues can be of any test type.
Code Block | ||
---|---|---|
| ||
{ "tests" : [ cucumberTestDefinition: { type"testKey" : "string" }DEMO-7", requirementKey: { type"start" : "string" 2013-05-03T11:47:35+01:00", }, "finish" labels: { : "2013-05-03T11:50:56+01:00", type: "arraycomment", : "Test was OK but the performance is items: {very poor", type"status" : "stringPASSED" }, } { } }, required: ["summarytestKey", "projectKey",: "cucumberTestDefinitionDEMO-8"], additionalProperties: false }; const TestCreateInfoGeneric = { id: "/TestCreateInfoGeneric", "start" type: "object2013-05-03T12:14:12+01:00", properties: { "finish" summary: {"2013-05-03T12:15:23+01:00", type"status" : "stringPASSED" }, projectKey: { "testKey" type: "stringDEMO-9", }, "start" genericTestDefinition: { : "2013-05-03T12:19:23+01:00", "finish" type: "string"2013-05-03T12:20:01+01:00", }, "comment" : "Error decreasing space requirementKey: { shuttle speed.", "status" type: "stringFAILED", "examples" }, labels: { type: "array",: [ "PASSED", "PASSED", "PASSED", "PASSED", "PASSED", "FAILED" ] items: {} ] } |
Importing manual test results with steps
This is a simple example of a JSON file with execution results for a manual test.
Code Block | ||
---|---|---|
| ||
{ type: "string" "tests" : [ } { } }, required: ["summarytestKey", "projectKey",: "genericTestDefinitionDEMO-57"], additionalProperties: false }; const ManualStep = { id: "/ManualStepstart", type: "object2014-08-30T12:19:23+01:00", properties: { "finish" action: { type: "string": "2014-08-30T12:20:01+01:00", }, "comment" : "Error executing data: {step 2!", type"status" : "stringFAILED", "steps": [ { }, result: { "status": "PASSED", type: "string" } }, required: ["action"], additionalProperties: false }; const Evidence = { "actualResult": "Step 1: OK" id: "/Evidence", type: "object"}, properties: { data: { type: "string" }"status": "FAILED", filename: { type"actualResult": "string"Step 2 *Failed* with an unexpected error message", }, contentType"evidences" : {[ type: "string" }{ }, required: ["data", "filename"], additionalProperties: false }; const ContextResult = { id: "/ContextResultdata", type: "object: "(... base 64 encoded ...)", properties: { name: { type"filename": "stringscreenshot1.jpg", }, duration: { type"contentType": "numberimage/jpeg" }, log: { // not being used } type: "string" }, status: {] type: "string", } ] } ] } |
Importing data-driven manual test results with auto-provisioning of tests
This is an example of a JSON file with a single test result.
This is a data-driven manual test with two iterations. For each iteration, we provide the parameters and the step results.
Xray will also create or update the test in Jira with the specification contained on the "testInfo" object.
Code Block | ||
---|---|---|
| ||
{enum: ["TODO", "FAILED", "PASSED", "EXECUTING"] },"tests": [ evidences: { type: "arraystart", items: { $ref: "/Evidence": "2021-08-30T11:47:35+01:00", } "finish" : "2021-08-30T11:50:56+01:00", }, "comment" examples: { "Successful execution", "status" type: "arrayPASSED", items"evidence" : {[ { type: "string", enum: ["TODOdata",: "FAILED", "PASSED", "EXECUTING"] iVBORw0KGgoAAAANSUhEUgAABkIAAAO9CAYAAADezXv6AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAEn(...base64 file enconding)", } }"filename": "image21.jpg", detailedExamples: { type"contentType": "arrayimage/jpeg", items: { } $ref: "/Example" ], "testInfo": }{ } }, required"summary": ["name", "status"]"Strong password validation", additionalProperties: false }; const Example = { id"type": "/ExampleManual", type: "object", properties: { "projectKey": "STORE", rank: { type"steps": "number"[ }, duration: { { type: "number" }, status"action": { "Open the Change Password screen by selecting option \"My Profile type: "string> Password\"", enum: ["TODO", "FAILED", "PASSED", "EXECUTING"] }"data": "", steps: { type"result": "array", items: { }, $ref: "/ExampleStep" }{ } }, required: ["status"], additionalProperties: false }; const ExampleStep = { id: "/ExampleStep""action": "Fill the password fields with data", type: "object", properties: { type: { "data": "Current Password: ${Password}\nNew Password: ${Password}\nConfirm New typePassword: "string${Password}", enum: ["hook", "background", "step"] }, keyword: { "result": "The new password is: ${Valid}\nError:\n${Message}" type: "string" }, rank: { ] type: "number" }, name: { "iterations": [ type: "string" },{ duration: { type"name": "numberIteration 1", }, errorMessage"parameters": {[ type: "string" { }, status: { type"name": "stringPassword", enum: ["TODO", "FAILED", "PASSED", "EXECUTING"] "value": "2635ftvu23v7t!09" }, evidences: { }, type: "array", { items: { $ref: "/Evidence" "name": "Valid", } } }, required: ["type", "keywordvalue",: "name", "status"], additionalProperties: false }; |
Example 1: Test with Test Type Gherkin or Unstructured
This is an example of a JSON file with execution results that will update Tests in Jira. The last item DEMO-9 must be a Test with a Test Type of Gherkin because the results contain examples.
Code Block | ||
---|---|---|
| ||
{ "tests" : [ Valid" }, { "testKeyname" : "DEMO-7Message", "startvalue" : "2013-05-03T11:47:35+01:00", "finish" : "2013-05-03T11:50:56+01:00", } "comment" : "Test was OK but the performance is very poor"], "statuslog" : "PASSEDPassword changed successfully", }, { "status": "PASSED", "testKeysteps" : "DEMO-8",[ "start" : "2013-05-03T12:14:12+01:00", { "finish" : "2013-05-03T12:15:23+01:00", "commentactualResult" : ""Performance, was better this time, in the context of test set DEMO-10.", "status" : "PASSED" }, {}, "testKey" : "DEMO-9", { "start" : "2013-05-03T12:19:23+01:00", "finishactualResult" : "2013-05-03T12:20:01+01:00Password changed successfully", "commentstatus" : "ErrorPASSED" decreasing space shuttle speed.", "status" : "FAILED", "examples" : [ "PASSED", "PASSED", "PASSED", "PASSED", "PASSED", "FAILED" } ] }, ] } |
Example 2: Tests with Test Type Step
This is an example of a JSON file with execution results that will update a Test with a Test Type of Step in Jira.
Code Block | ||
---|---|---|
| ||
{ { "name": "Iteration 2", "testsparameters" : [ { "testKeyname" : "DEMO-57Password", "start" : "2014-08-30T12:19:23+01:00", "finishvalue" : "2014-08-30T12:20:01+01:00",123123" "comment" : "Error executing step 2!", }, "status" : "FAILED", "steps": [ { "statusname": "PASSEDValid", "commentvalue": "Step 1: OKNot Valid" }, }, { { "status": "FAILED", "commentname": "StepMessage", 2 *Failed* with an unexpected error message", "evidencesvalue" : [: "Password is too simple." } { ], "datalog": "(... base 64 encoded ...)Password validation check failed. Password too simple!", "filename"status": "screenshot1.jpgFAILED", "contentType"steps": "image/jpeg"[ { } ]"actualResult": "", } ] } ] } |
Importing Multiple Execution results
Xray supports importing multiple results for the same Test issue in the same execution. These results often indicate different contexts/environments where the same Test was executed. Xray will group all executions of the same Test in a single Test Run and present all execution information, including the different contexts in the Execution page.
Tip |
---|
When importing execution results using the Cucumber JSON output formatter, multiple executions for the same scenario/Test issue will be merged into a single Test Run. When importing execution results using the Xray JSON format, the results JSON element must contain all execution results for the same Test Run. |
Xamarin Test Cloud
If you are using the Xamarin Test Cloud for executing Cucumber mobile Tests in different combinations of mobile devices and operating systems, you can import the results to Jira by making a compressed zip file containing the multiple Cucumber JSON files.
Info | ||
---|---|---|
| ||
For more information, please refer to Testing using Calabash and Xamarin Test Cloud in Ruby and the "bundle" endpoint in the REST API."status": "PASSED"
},
{
"actualResult": "Password too simple!",
"status": "FAILED"
}
]
}
]
}
]
} |