Page History
...
External execution results from either automated and manual Tests can be imported to Jira. This operation may be done in one of two ways:
- Manually, using the Import Execution Results action accessible from the Test Execution issue screen
- via the REST API to integrate with Continuous Integration (CI) platforms or other external execution processes. Please refer to the Xray REST API.
...
testExecutionKey | The test execution key where to import the execution results |
info | The info object for creating new Test Execution issues (link) |
tests | The Test Run result details (link) |
addTestsToPlan | Whether the tests must be added to the Test Plan as well can impact the performance. (link) |
"info" object - Test Execution issue
...
It is possible to import a single result (the test object itself with the "steps" (Manual tests) or "examples" (BDD tests)) or multiple execution results into the same Test Run (data-driven testing) using the "iterations" array.
If you have Xray Enterprise installed, if the "testVersion" is provided Xray will import the results using the provided test version as the base for the Test Run. Otherwise, Xray will try to resolve the imported execution result into an active test version. The Default version will be chosen if determined to be of the same Test Type. If not, Xray will use the latest active test version if the same Test Type.
testKey | The test issue key |
testInfo | The testInfo element (link) |
testVersion | The Test Version to import the result |
start | The start date for the test run |
finish | The finish date for the test run |
comment | The comment for the test run |
executedBy | The user id who executed the test run |
assignee | The user id for the assignee of the test run |
status | The test run status (PASSEDPASS, FAILEDFAIL, EXECUTING, TODO, custom statuses ...) |
steps | The step results (link) |
examples | The example results for BDD tests (link) |
iterations | The iteration containing data-driven test results (link) |
defects | An array of defect issue keys to associate with the test run |
evidence | An array of evidence items of the test run (link) |
customFields | An array of custom fields for the test run (link) |
...
Xray will first try to match test issues by the testKey if present. Otherwise, Manual or BDD tests are matched by summary whilst Generic tests are matched using the generic definition field, within the same project.
If Xray Enterprise is installed, and if the Test issue is identified by testKey or summary, and if the Test Type of the testInfo element does not exist in any version of the Test, then a new version will be created automatically for the new definition. Otherwise, Xray will update the existing Test version using the default if the Test Type matches or the latest active version of the same Test Type.
Any changes to the testInfo element will update the test issue specification in Jira.
If the match field (summary or definition) is changed, Xray will search for another issue and will create a new test case, or update an existing test case if no one is found. If you need to change the summary or the definition, you can do it manually (go to Jira and change the field), or you can include the testKey within the test element.
...
name | The iteration name |
parameters | An array of parameters along with their values (link) |
status | The status for the iteration (PASSEDPASS, FAILEDFAIL, EXECUTING, TODO, custom statuses ...) |
...
Code Block | ||||
---|---|---|---|---|
| ||||
{ "$id": "XraySchema", "type": "object", "properties": { "testExecutionKey": { "type": "string" }, "info": { "type": "object", "properties": { "project": { "type": "string" }, "summary": { "type": "string" }, "description": { "type": "string" }, "version": { "type": "string" }, "revision": { "type": "string" }, "user": { "type": "string" }, "startDate": { "type": "string", "format": "date-time" }, "finishDate": { "type": "string", "format": "date-time" }, "testPlanKey": { "type": "string" }, "testEnvironments": { "type": "array", "items": { "type": "string" } } }, "additionalProperties": false }, "tests": { "type": "array", "items": { "$ref": "#/definitions/Test" }, "minItems": 1, } }, "additionalProperties": false, "definitions": { "Test": { "type": "object", "properties": { "testKey": { "type": "string" }, "testInfo": { "$ref": "#/definitions/TestInfo" }, "start": { "type": "string", "format": "date-time" }, "finish": { "type": "string", "format": "date-time" }, "comment": { "type": "string" }, "executedBy": { "type": "string" }, "assignee": { "type": "string" }, "status": { "type": "string" }, "steps": { "type": "array", "items": { "$ref": "#/definitions/ManualTestStepResult" } }, "examples": { "type": "array", "items": { "type": "string", "enum": ["TODO", "FAILEDFAIL", "PASSEDPASS", "EXECUTING"] } }, "iterations": { "type": "array", "items": { "$ref": "#/definitions/IterationResult" } }, "defects": { "type": "array", "items": { "type": "string" } }, "evidence": { "type": "array", "items": { "$ref": "#/definitions/EvidenceItem" } }, "evidences": { // DEPRECATED "type": "array", "items": { "$ref": "#/definitions/EvidenceItem" } }, "customFields": { "$ref": "#/definitions/CustomField" } }, "required": ["status"], "dependencies": { "evidence": { "not": { "required": ["evidences"] } }, "evidences": { "not": { "required": ["evidence"] } }, "steps": { "allOf": [ { "not": { "required": ["examples"] } }, { "not": { "required": ["iterations"] } } ] }, "examples": { "allOf": [ { "not": { "required": ["steps"] } }, { "not": { "required": ["iterations"] } } ] }, "iterations": { "allOf": [ { "not": { "required": ["steps"] } }, { "not": { "required": ["examples"] } } ] } }, "additionalProperties": false }, "IterationResult": { "type": "object", "properties": { "parameters": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "value": { "type": "string" } }, "additionalProperties": false } }, "status": { "type": "string" }, "steps": { "type": "array", "items": { "$ref": "#/definitions/ManualTestStepResult" } } }, "required": ["status"], "additionalProperties": false }, "ManualTestStepResult": { "type": "object", "properties": { "status": { "type": "string" }, "comment": { "type": "string" }, "evidences": { "type": "array", "items": { "$ref": "#/definitions/EvidenceItem" } }, "defects": { "type": "array", "items": { "type": "string" } }, "actualResult": { "type": "string" } }, "required": ["status"], "additionalProperties": false }, "TestInfo": { "type": "object", "properties": { "summary": { "type": "string" }, "description": { "type": "string" }, "projectKey": { "type": "string" }, "requirementKeys": { "type": "array", "items": { "type": "string" } }, "labels": { "type": "array", "items": { "type": "string" } }, "testType": { "type": "string" }, steps: { type: "array", items: { type: "object", properties: { action: { type: "string" }, data: { type: "string" }, result: { type: "string" } }, required: ["action"], additionalProperties: false } }, "scenario": { "type": "string" }, "scenarioType": { "type": "string" }, "definition": { "type": "string" } }, "dependencies": { "steps": { "allOf": [ { "not": { "required": ["scenario"] } }, { "not": { "required": ["definition"] } } ] }, "scenario": { "allOf": [ { "not": { "required": ["steps"] } }, { "not": { "required": ["definition"] } } ] }, "definition": { "allOf": [ { "not": { "required": ["steps"] } }, { "not": { "required": ["scenario"] } } ] } }, "required": ["summary", "projectKey", "testType"], "additionalProperties": false }, "EvidenceItem": { "type": "object", "properties": { "data": { "type": "string" }, "filename": { "type": "string" }, "contentType": { "type": "string" } }, "required": ["data", "filename"], "additionalProperties": false }, "CustomField": { "type": "array", "items": { "type": "object", "properties": { "id": { "type": "string" }, "value": {} }, "required": ["id", "value"], "additionalProperties": false } } } } |
...
Code Block | ||
---|---|---|
| ||
{ "tests" : [ { "testKey" : "DEMO-57", "start" : "2014-08-30T12:19:23+01:00", "finish" : "2014-08-30T12:20:01+01:00", "comment" : "Error executing step 2!", "status" : "FAILEDFAIL", "steps": [ { "status": "PASSEDPASS", "actualResult": "Step 1: OK" }, { "status": "FAILEDFAIL", "actualResult": "Step 2 *Failed* with an unexpected error message", "evidences" : [ { "data": "(... base 64 encoded ...)", "filename": "screenshot1.jpg", "contentType": "image/jpeg" } ] } ] } ] } |
Example 4: Importing test results with Test Run Custom Fields
This is a simple example of a JSON file with execution results with Test Run Custom Fields values.
Code Block | ||
---|---|---|
| ||
{ "info": { "summary": "Execution of automated tests for release v1.3", "description": "This execution is automatically created when importing execution results from an external source", "user": "admin", "startDate": "2014-08-30T11:47:35+01:00", "finishDate": "2014-08-30T11:53:00+01:00" }, "tests": [ { "testKey": "DEMO-25", "start": "2014-08-30T11:47:35+01:00", "finish": "2014-08-30T11:50:56+01:00", "comment": "Successful execution", "status": "FAIL", "customFields": [ { "id": 1, "value": "true" }, { "id": 2, "value": "13400" }, { "id": 3, "value": "Test instance ] } ] } is refreshed" } ] } ] } |
Example
...
5: Importing data-driven manual test results with auto-provisioning of tests
This is an example of a JSON file with a single test result.
...
Code Block | ||
---|---|---|
| ||
{ "tests": [ { "start" : "2021-08-30T11:47:35+01:00", "finish" : "2021-08-30T11:50:56+01:00", "comment" : "Successful execution", "status" : "PASSEDPASS", "evidence" : [ { "data": "iVBORw0KGgoAAAANSUhEUgAABkIAAAO9CAYAAADezXv6AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAEn(...base64 file enconding)", "filename": "image21.jpg", "contentType": "image/jpeg" } ], "testInfo": { "summary": "Strong password validation", "type": "Manual", "projectKey": "STORE", "steps": [ { "action": "Open the Change Password screen by selecting option \"My Profile > Password\"", "data": "", "result": "" }, { "action": "Fill the password fields with data", "data": "Current Password: ${Password}\nNew Password: ${Password}\nConfirm New Password: ${Password}", "result": "The new password is: ${Valid}\nError:\n${Message}" } ] }, "iterations": [ { "parameters": [ { "name": "Password", "value": "2635ftvu23v7t!09" }, { "name": "Valid", "value": "Valid" }, { "name": "Message", "value": "" } ], "status": "PASSEDPASS", "steps": [ { "actualResult": "", "status": "PASSEDPASS" }, { "actualResult": "Password changed successfully", "status": "PASSEDPASS" } ] }, { "parameters": [ { "name": "Password", "value": "123123" }, { "name": "Valid", "value": "Not Valid" }, { "name": "Message", "value": "Password is too simple." } ], "status": "FAILEDFAIL", "steps": [ { "actualResult": "", "status": "PASSEDPASS" }, { "actualResult": "Password too simple!", "status": "FAILEDFAIL" } ] } ] } ] } |
Importing Multiple Execution results
...