Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
 

Overview

Xray enables easy integration with Jenkins through the "Xray for JIRA Jenkins Plugin", providing the means for successful Continuous Integration by allowing users to report automated testing results.

...

The installation is made manually. For more information on how to install add-ons, please refer to to how to install add-ons.


Info
titleRequirements

The Jenkins baseline for this app is v2.138.4 and it may not work properly with previous versions.

...

Xray for Jenkins is configured in the global settings configuration page  Manage Jenkins >  > Configure System > Xray > Xray for Jira configuration.

Jira servers

The Jira servers configuration defines connections with Jira instances. 

To add a new Jira instance connection, you need to specify some properties:

  1. Configuration alias
  2. Hosting:  Hosting (instance type) in this case Server/Data Center.
  3. Server Address: The address of the Jira Server where Xray is running
  4. Credentials:
    1. Use the Jenkins Credentials PluginCredentials Plugin to set the username/password (if you are using a Server/Data Center instance).
    2. Make sure that the user you are using have the following permissions in the projects where you want to import the results and import/export feature files: View, Edit, Create.
    3. This field is optional - if you don't want to use a System scoped credential to authenticate in your instance, you can leave this field empty and force the users to use an User scoped credential in the build task.

...

This build step will export the Cucumber Tests (i.e., Scenario/Scenario Outlines) in .feature or bundled in a .zip file. The rules for exporting are defined defined here.

It invokes Xray's Export Cucumber Tests REST API endpoint (see more information here).

...

Some fields need to be configured in order to export the Cucumber Tests. As input, you can either specify issue keys (see the endpoint documention documention here) or the ID of the saved filter in Jira.

...

fielddecription
JIRA instanceThe Jira instance where Xray is running.
CredentialsIf the above Jira Instance does not have any credential configured, you must define an User scoped credential here
Project KeyThis is the project where the Tests and Pre-Conditions will be created/updated.
Cucumber feature files directoryThis is the directory containing your feature files. All the files in this directory and sub directories will be imported. Supports both relative and absolute paths.
Modified in the last hoursBy entering an integer integer  n here here, only files that where modified in the last last  n hours  hours will be imported. 
Leave empty if you do not want to use this parameter.

...

For those formats, the file path needs to be relative to the workspace.

ConfigurationConfiguration 

fielddescription
Jira instanceThe Jira instance where Xray is running
CredentialsIf the above Jira Instance does not have any credential configured, you must define an User scoped credential here
FormatA list of test result formats and its specific endpoint
Execution Report File

The results relative or absolute file path

Note: glob expressions are supported for

  • JUnit
  • JUnit Multipart
  • TestNG
  • TestNG Multipart
  • NUnit
  • NUnit Multipart
  • Robot framework
  • Robot framework Multipart

...

Format and specific endpointFieldDescription

Behave JSON multipart

Cucumber JSON multipart

NUnit XML multipart

JUnit XML multipart

Robot XML multipart

TestNG XML multipart

Import to Same Test ExecutionWhen this option is check, if you are importing multiple execution report files using a glob expression, the results will be imported to the same Test Execution

Test execution fields

An object (JSON) specifying the fields for the issue. You may specify the object either directly in the field or in the file path.



Info
titleLearn more

The custom field IDs can be obtained be obtained using the Jira REST API Browser tool included in Jira. Each ID is of the form "customfield_ID".

Another option, which does not require Jira administration rights, is  is to invoke the "Get edit issue meta" in an existing issue (e.g., in a Test issue) as mentioned mentioned here.

Example: GET GET  http://yourserver/rest/api/2/issue/CALC-1/editmeta 

Import in parallelIf there are several result files, when this checkbox is selected, we will import all the files in parallel (using all available CPU cores)


NUnit XML

JUnit XML

Robot XML

TestNG XML

Import to Same Test ExecutionWhen this option is check, if you are importing multiple execution report files using a glob expression, the results will be imported to the same Test Execution
Project keyKey of the project where the Test Execution (if the the T est Execution Key field  field wasn't provided) and the Tests (if they aren't created yet) are going to be created
Test execution key

Key of the Test Execution

Test plan keyKey of the Test Plan
Test environmentsList of Test Environments separated by ";"
RevisionSource code's revision being target by the Test Execution
Fix versionThe Fix Version associated with the test execution (it supports only one value)
Import in parallelIf there are several result files, when this checkbox is selected, we will import all the files in parallel (using all available CPU cores)

...

Info
titlePipeline Project Limitations

Due to Jenkins limitations, these variables will not be set on Pipeline projects.


Examples

Cucumber

In a typical typical Cucumber Workflow, after having created a Cucumber project and the Cucumber tests specified in Jira, you may want to have a project that exports the features from Jira, executes the automated tests on a CI environment and environment and then imports back its results.

...

In this example, we configured the task to extract the the features from a set of issues (PROJ-78 and PROJ-79) to the folder that holds the Cucumber project.

...

To start the configuration, add the build step Xray Xray: Cucumber Features Import Task.

...

To start the configuration, add the post-build action Xray Xray: Results Import Task.

After that, configure it.

In this example, we  we configured the task to import the Cucumber JSON results back to Jira.

...

Once all configurations are done, click Save at the bottom of the page.

After running the jobthe job, the expected result is a new Test Execution issue created in the Jira instance.

...

For Cucumber, Behave, JUnit, Nunit and Robot, Xray for Jenkins allows you to create new Test Executions and have control over newly-created Test Execution fields. You can send two files, the normal execution result file and a JSON file similar to the one Jira one Jira uses to create new issues. More details regarding how Jira creates new issues here. 

For this scenario and example, the import task needs to be configured with the Cucumber JSON Multipart format. When selecting this option, you can additionally configure the Test Execution fields in  in one of two ways:

  • Insert the relative path to the JSON file containing the information, or
  • Insert the JSON content directly in the field.

...

And configured the task to import the the Cucumber JSON Multipart results  results back to Jira.

Once all configurations are done, click Save at the bottom of the page.

...

Apart from supporting Cucumber natively, Xray enables you enables you to take advantage of many other testing frameworks like JUnit. In this sense, Xray for Jenkins lets you import results in other formats besides Cucumber JSON.

...

To start the configuration, add the post-build action Xray Xray: Results Import Task.

After that, configure it.

...

There are 3 available steps to be used in a Pipeline project:

  • XrayImportBuilder - Import test results (Junit, NUnit, etc...) from your Jenkins job to Jira
  • XrayExportBuilder- Export feature files from Jira to your Jenkins job workspace

  • XrayImportFeatureBuilder - Import feature files from Jenkins to Jira

...

Expand
titleParameters
ParameterRequired?TypeDescription
serverInstanceYesStringThe ID of the Jira instance configured in the Jenkins System Configuration
endpointNameYesString

The result file type to be imported. Allowed Values:

  • "" (Xray Json format)
  • "/multipart"  (Xray JSON multipart format)
  • "/cucumber"
  • "/cucumber/multipart"
  • "/behave"
  • "/behave/multipart"
  • "/junit"
  • "/junit/multipart"
  • "/nunit"
  • "/nunit/multipart"
  • "/robot"
  • "/robot/multipart"
  • "/bundle" (import of zip file with several cucumber results)
  • "/testng"
  • "/testng/multipart"
  • "/xunit"
  • "/xunit/multipart"

Please note that not all endpoints are available for Jira Sever/Cloud. Please refer to the REST API documentation to see what is available in your instance.

projectKeyYesStringThe Jira project key where you want to import your results
importFilePathYesString

File path where the result files can be found.

credentialIdYes, if the Jira instance was configured without credentials in System ConfigurationStringCredential ID from the Credentials plugin to be used to authenticate the Jira requests
testEnvironmentsNoString

Test enviroments to be added to the Test Execution issue, seperated by ";".

(warning) This value will only be used if the endpointName is not multipart

testPlanKeyNoString

All Tests will be added to the given Test Plan key, if provided.

(warning) This value will only be used if the endpointName is not multipart

fixVersionNoString

Fix version to be added to the Test Execution issue.

(warning) This value will only be used if the endpointName is not multipart

testExecKeyNoString

Key of the test Exeution issue to be updated. Leave empty to create a new issue with the import.

(warning) This value will only be used if the endpointName is not multipart

revisionNoString

Source code and documentation version used in the test execution.

(warning) This value will only be used if the endpointName is not multipart

importInfoYes, if multipart endpointStringFile path to the Test Execution info file OR JSON String with the info.
inputInfoSwitcherYes, if importInfo is being usedString

Allowed values:

  • "filePath" - if importInfo field is used and represents a file path
  • "fileContent" - if importInfo field is used and represents a JSON text
testImportInfoNoStringFile path to the Test info file.
inputTestInfoSwitcherYes, if testImportInfo is being usedString

Allowed values:

  • "filePath" - if if testImportInfo field is used and represents a file path
  • "fileContent" - if if testImportInfo field is used and represents a JSON text
importToSameExecutionNoString

Allowed values:

  • "true" - to import all created tests and linked them to a single Test Execution issue
  • "" - link a Test Execution issue to every imported Test issue
importInParallelNoString

Allowed values:

  • "true" - to import all result files (if there are multiple) in parallel, in order to speed up the import process
  • "" - to import all result files (if there are multiple) sequentially

...

Expand
titleExample: Nunit multipart result import with Test info and using environment variables
Code Block
languagegroovy
        stage('Import results to Xray (multipart)') {
            steps {
                step([$class: 'XrayImportBuilder', endpointName: '/nunit/multipart', importFilePath: '/reports/*.xml', importToSameExecution: 'true', projectKey: 'CALC', serverInstance: 'ecc67055-c359-40cb-8b8a-a44cb9f6ca30', importInParallel: 'true', importInfo: '/info/my-test-exec-info.json', inputInfoSwitcher: 'filePath', inputTestInfoSwitcher: 'fileContent', testImportInfo: '''{
           "fields": {
              "project": {
                 "key": "CALC"
              },
              "summary": "Test Execution for java junit ${BUILD_NUMBER}",
              "issuetype": {
                 "id": "9"
              },
              "customfield_11807": [
                 "CALC-1200"
              ]
           }
        }'''])
            }
        }
Info

importInfo and testImportInfo must comply with the same format as the Jira issue create/update REST API format

Step: XrayExportBuilder  XrayExportBuilder (export cucumber features from Jira from Jira to Jenkins)


Expand
titleParameters
ParameterRequired?TypeDescription
serverInstanceYesStringThe ID of the Jira instance configured in the Jenkins System Configuration
issues

Yes

(not required if filter is used)

String

Xray Tests/Test Plans/Test Sets/Test Execution issue keys, seperated by ';'.

filter

Yes

(not required if issues is used)

StringThe Jira filter ID containing Xray Tests/Test Plans/Test Sets/Test Execution issues
filePathNoString

The default value is "/features"

File path where the feature files will be downloaded

credentialIdYes, if the Jira instance was configured without credentials in System ConfigurationStringCredential ID from the Credentials plugin to be used to authenticate the Jira requests

...

Expand
titleExample: Export features using variables
Code Block
languagegroovy
        stage('Export feature files') {
            step([$class: 'XrayExportBuilder', issues: '${MY_ISSUE_KEYS}', filter: '${MY_FILTER_ID}', filePath: '${MY_FILE_PATH}', serverInstance: 'ecc67055-c359-40cb-8b8a-a44cb9f6ca30', credentialId: '26dba0be-45ca-4ffd-b959-13dbd2410a82'])
        }

Step: XrayImportFeatureBuilder  XrayImportFeatureBuilder (import cucumber features from Jenkins to Jira)

...

This is a declarative example, for Cucumber tests using the "VCS/Git based" workflow (see see Testing in BDD with Gherkin based frameworks (e.g. Cucumber)).

...

This is the simplest way to generate your step script, and we strongly recommend the use of this snippet due to the complexity of some task related parameters. 

Jira instances configuration via Groovy script (Jenkins Script Console)

...

When you check the log, it has the following:

By default, when you successively uccessively try to log into Jira with the wrong credentials, the Jira instance will prompt you to provide a CAPTCHA the next time you try to log in. It is not possible to provide this information via the build process, so it will fail with status code 403 Forbidden.

...