Image Added

Overview

Xray enables easy integration with TeamCity through the "Xray TeamCity Plugin", providing the means for successful Continuous Integration by allowing users to report automated testing results.

Release Notes

Installation

The installation is made manually. For more information on how to install add-ons, please refer to how to install add-ons.

Manual Installation

1. Upload the plugin
   
   

2. Restart TeamCity (if needed as shown in the screenshot below)

Image Added

Configuration

Xray for TeamCity is configured in the global settings configuration page  Administration > Integrations > Xray.

Jira servers

The Jira servers configuration defines connections with Jira instances. 

To add a new Jira instance connection, you need to specify some properties:

1. Configuration alias: a friendly name for the configuration
2. Server or Cloud: Server
   
3. Server Address: The address of the Jira Server where Xray is running
4. Authentication:
   
   1. User: username
   2. Password.

Jira Cloud

The Jira Cloud configuration defines connections with Jira Cloud. 

To add a new Jira Cloud connection, you need to specify some properties:

1. Configuration alias: a friendly name for the configuration
2. Server or Cloud: Cloud
3. Authentication:
   
   1. Client ID: obtained from Xray Cloud (more info here)
   2. Client Secret: obtained from Xray Cloud (more info here)

Image Added

Build Steps

Build steps are the building blocks of the build process. These need to be defined in the build configuration.

The app provides:

• one build step for exporting Cucumber Scenario/Scenario Outlines from Jira as .feature files
• one build step for importing Cucumber Tests from existing Cucumber features into Jira.
• one build step which publishes the execution results back to Jira.

Xray: Cucumber Features Export Task

This build step will export the Cucumber Tests (i.e., Scenario/Scenario Outlines) in .feature or bundled in a .zip file. The rules for exporting are defined here.

It invokes Xray's Export Cucumber Tests REST API endpoint (see more information: server or cloud).

Configuration

Some fields need to be configured in order to export the Cucumber Tests. As input, you can either specify issue keys (see the endpoint documentation for the server or cloud) or the ID of the saved filter in Jira.

Xray: Cucumber Features Import Task

This build step is only available for the server and will import existing cucumber Tests from existing Cucumber feature files into Xray issues. This Task will import from .feature files and also from .zip files.

It invokes Xray's Import Cumcumber Tests REST API endpoint (see more information here)

Xray: Results Import Task

The app provides easy access to Xray's Import Execution Results REST API endpoints (see more information for the server or cloud). Therefore, it mimics the endpoints input parameters.

It supports importing results in Xray's own JSON format, Cucumber, Behave, JUnit, and NUnit, among others. However, there are some endpoints in the server that may not yet be available for Xray Cloud. 

These are the available endpoints.

	Server	Cloud
Xray JSON
Cucumber JSON
Cucumber JSON multipart
Behave JSON
Behave JSON multipart
JUnit XML
JUnit XML multipart
NUnit XML
NUnit XML multipart
Robot XML
Robot XML multipart
Compressed .zip file
TestNG XML
TestNG XML multipart

Using a glob expression, you can import multiple results files in the following formats:

• JUnit
• TestNG
• NUnit
• Robot framework

For those formats, the file path needs to be relative to the workspace.

Configuration 

Additional fields

Depending on the chose test result format and endpoint, some additional fields may need to be configured.

Examples

Cucumber

In a typical Cucumber Workflow, after having created a Cucumber project and the Cucumber tests specified in Jira, you may want to have a project that exports the features from Jira, executes the automated tests on a CI environment and then imports back its results.

For this scenario, the Jenkins project would be configured with a set of tasks responsible for:

1. Pulling the Cucumber project
2. Exporting Cucumber features from Jira to your Cucumber project
3. Executing the tests in the CI environment
4. Importing the execution results back to Jira

Exporting Cucumber features

To start the configuration, add the build step Xray: Cucumber Features Export Task.

Image Added

After that, configure it.

In this example, we configured the task to extract the features from a set of issues (PROJ-78 and PROJ-79) to the folder that holds the Cucumber project.

Image Added

Importing Cucumber features

To start the configuration, add the build step Xray: Cucumber Features Import Task.

Image Added

After that, configure it.

In this example, we configured the task to import to the Project TEAM of the Xray instance all the .features and .zip files that are contained in /Cucumber directory and sub directories, which were modified in the last 3 hours.

Image Added

Importing the execution results

To start the configuration, add a build step Xray: Results Import Task.

Image Added

After that, you may configure it.

In this example, we configured the task to import the Cucumber JSON results back to Jira.

Image Added

Once all configurations are done, click Save at the bottom of the page.

After running the job, the expected result is a new Test Execution issue created in the Jira instance.

Image Added

Importing the execution results with user-defined field values

For Cucumber, Behave, JUnit, Nunit and Robot, Xray for Jenkins allows you to create new Test Executions and have control over newly-created Test Execution fields. You can send two files, the normal execution result file and a JSON file similar to the one Jira uses to create new issues. More details regarding how Jira creates new issues here. 

For this scenario and example, the import task needs to be configured with the Cucumber JSON Multipart format. When selecting this option, you can additionally configure the Test Execution fields in one of two ways:

• Insert the relative path to the JSON file containing the information;
• Or, insert the JSON content directly in the field.
  
  

In this example, we configured the following object:

{
   "fields": {
      "project": {
         "key": "PROJ"
      },
      "summary": "Test Execution for Cucumber results",
      "issuetype": {
         "id": "10102"
      }
   }
}

And configured the task to import the Cucumber JSON Multipart results back to Jira.

Image Added

Once all configurations are done, click Save.

After running the job, the expected result is a new Test Execution issue created in the Jira instance, with the Test Execution fields as specified in the build step configuration.

JUnit

Apart from supporting Cucumber natively, Xray enables you to take advantage of many other testing frameworks like JUnit. In this sense, Xray for Jenkins lets you import results in other formats besides Cucumber JSON.

If you want to import JUnit XML reports, a typical Job outline would be:

1. Pulling the JUnit project
2. Executing the tests in the CI environment
3. Importing the execution results, including Tests, to JIRA

Importing the execution results

To start the configuration, add the build step Xray: Results Import Task.

Image Added

After that, configure it.

In this example, we have a configuration where the JUnit XML format is chosen.

After running the plan, the expected result is a new Test Execution issue created in the JIRA instance.

Make sure to select "Always, even if build stop command was issued" so, even some previous step fail, the tests will still be imported.

Image Added

Image Added

You can also import multiple results using a glob expression, like in the following example

Image Added

Troubleshooting

The build process is failing with status code 403

When you check the log, it has the following:

Image Added

By default, when you successively try to log into Jira with the wrong credentials, the Jira instance will prompt you to provide a CAPTCHA the next time you try to log in. It is not possible to provide this information via the build process, so it will fail with status code 403 Forbidden.

You will need to log into Jira via the browser and provide the CAPTCHA.

Image Added

If you are a Jira administrator, you can go to Jira administration > User Management and reset the failed login.

Image Added

Invalid JIRA instance

When you check the log, it has the following:

Image Added

This occurs when a JIRA instance is selected in the build step configuration and that instance is later deleted in the Xray global settings.

This error is shown in the build step.

Image Added

To solve this, edit the build step, select a valid JIRA Instance and save.

Xray does not provide yet a plugin for TeamCity. However, it is easy to setup TeamCity in order to integrate it with Xray.

Since Xray provides a full REST API, you may interact with Xray, for submitting results for example.

JUnit example

In this scenario, we want to get visibility of the automated test results from some tests implemented in Java, using the JUnit framework. 

This recipe could also be applied for other frameworks such as NUnit or Robot.

package com.xpand.java;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.assertThat;

public class CalcTest {

    @Before
    public void setUp() throws Exception {

    }

    @After
    public void tearDown() throws Exception {

    }

	@Test
    public void CanAddNumbers()
    {
        assertThat(Calculator.Add(1, 1), is(2));
        assertThat(Calculator.Add(-1, 1), is(0));
    }


    @Test
    public void CanSubtract()
    {
        assertThat(Calculator.Subtract(1, 1), is(0));
        assertThat(Calculator.Subtract(-1, -1), is(0));
        assertThat(Calculator.Subtract(100, 5), is(95));
    }


    @Test
    public void CanMultiply()
    {
        assertThat(Calculator.Multiply(1, 1), is(1));
        assertThat(Calculator.Multiply(-1, -1), is(1));
        assertThat(Calculator.Multiply(100, 5), is(500));
    }


    public void CanDivide()
    {
        assertThat(Calculator.Divide(1, 1), is(1));
        assertThat(Calculator.Divide(-1, -1), is(1));
        assertThat(Calculator.Divide(100, 5), is(20));
    }


    @Test
    public void CanDoStuff()
    {
        assertThat(true, is(true));
    }


}

In order to submit those results, we'll just need to invoke the REST API (as detailed in Import Execution Results - REST).

Run automated tests

Our project is Maven based, therefore the first Build Step compiles and runs the JUnit automated tests.

Image Removed

Image Removed

Import execution results

In order to submit the results, we'll need to add a Build Step of type "Command Line", where we'll invoke the REST API, submitting the JUnit XML report generated in the previous step.

Image Removed

The complete script content of the "custom script" field above is:

curl -H "Content-Type: multipart/form-data" -u %jira_user%:%jira_password% -F "file=@java-junit-calc/target/surefire-reports/TEST-com.xpand.java.CalcTest.xml" "%jira_base_url%/rest/raven/1.0/import/execution/junit?projectKey=CALC&fixVersion=v3.0&revision=1234"

We're using "curl" utility that comes in Unix based OS'es but you can easily use another tool to make the HTTP request.

Notice that we're using some parameters for storing Jira's base URL along with the credentials to be used in the REST API.

Actually, these parameters can be defined at multiple levels; in our example we defined them at the "Build Configuration" level but they could also have been defined at  the project level. 

Image Removed

The parameters can be hidden, such as the password, if you defined them as being of type "Password". 

Image Removed

Cucumber example

In this scenario, we are managing the specification of Cucumber Scenarios/Scenario Outline(s) based tests in Jira, as detailed in the "standard workflow" mentioned in Testing with Cucumber

Then we need to extract this specification from Jira (i.e. generate related Cucumber .feature files), and run it in TeamCity against the code that actually implements each step that are part of those scenarios.

Finally, we can then submit the results back to JIRA and they'll be reflected on the related entities. 

Overall, our Build Configuration is composed of 3 basic steps.

Image Removed

Exporting Cucumber features 

We start by extracting the tests specification out of JIRA and generate the proper .feature files.

The export can take as input issue keys of requirements, Test Executions, Test Plans or a filter id, which will be the one we'll use.

For this, we'll invoke the REST API (Exporting Cucumber Tests - REST) in order to obtain a .zip file containing the .feature files.

We'll be using a Build Step of type "Command Line" for this purpose, along with "curl" utility to ease making the HTTP request.

Image Removed

The complete script content of the "custom script" field above is:

curl -u %jira_user%:%jira_password% "%jira_base_url%/rest/raven/1.0/export/test?filter=11400&fz=true" -o features/features.zip
unzip -o features/features.zip -d features/

Notice that we're unzipping the .feature files to a local directory, so we're able to run them.

Run Cucumber scenarios

The exact syntax for running the Cucumber scenarios depends on the Cucumber implementation being used; in this case we're using Ruby's variant.

Therefore we're basically just invoking "cucumber" command with an option to generate a JSON report (e.g. "data.json").

Image Removed

You may have noticed a trick in the cucumber line above, in the end of the command (i.e. ".... || :"). That ensures that cucumber returns with exit code 0 (i.e. success), so the build may proceed.

Import execution results

In order to submit the results, we'll need to add a Build Step of type "Command Line", where we'll invoke the REST API, submitting the Cucumber JSON report generated in the previous step.

We also make sure this step is called always.

Image Removed

The complete script content of the "custom script" field above is:

curl -v -H "Content-Type: application/json" -X POST -u %jira_user%:%jira_password% --data @data.json "%jira_base_url%/rest/raven/1.0/import/execution/cucumber"

You may notice that we're using some parameters related with the Jira server, that we've configured at project level.

Image Removed