CircleCI is a well-known CI/CD tool available on-premises and as SaaS.

Xray does not provide yet a plugin for CircleCI. However, it is easy to setup CircleCI in order to integrate it with Xray.

Since Xray provides a full REST API, you may interact with Xray, for submitting results for example.


JUnit example

In this scenario, we want to get visibility of the automated test results from some tests implemented in Java, using the JUnit framework. 

This recipe could also be applied for other frameworks such as NUnit, TestNG or Robot.

We need to setup a project based on a Git repository containing the code along with the configuration for CircleCI build process.


The tests are implemented in a JUnit class as follows.

CalcTest.java
package com.xpand.java;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.assertThat;

public class CalcTest {

    @Before
    public void setUp() throws Exception {

    }

    @After
    public void tearDown() throws Exception {

    }

	@Test
    public void CanAddNumbers()
    {
        assertThat(Calculator.Add(1, 1), is(2));
        assertThat(Calculator.Add(-1, 1), is(0));
    }


    @Test
    public void CanSubtract()
    {
        assertThat(Calculator.Subtract(1, 1), is(0));
        assertThat(Calculator.Subtract(-1, -1), is(0));
        assertThat(Calculator.Subtract(100, 5), is(95));
    }


    @Test
    public void CanMultiply()
    {
        assertThat(Calculator.Multiply(1, 1), is(1));
        assertThat(Calculator.Multiply(-1, -1), is(1));
        assertThat(Calculator.Multiply(100, 5), is(500));
    }


    public void CanDivide()
    {
        assertThat(Calculator.Divide(1, 1), is(1));
        assertThat(Calculator.Divide(-1, -1), is(1));
        assertThat(Calculator.Divide(100, 5), is(20));
    }


    @Test
    public void CanDoStuff()
    {
        assertThat(true, is(true));
    }


}


The CircleCI configuration file .circleci/config.yml contains the definition of the build steps, including running the automated tests and submitting the results.

.circleci/config.yml
version: 2 # use CircleCI 2.0
jobs: # a collection of steps
  build: # runs not using Workflows must have a `build` job as entry point

    working_directory: ~/demo/java-junit-calc # directory where steps will run

    docker: # run the steps with Docker
      - image: circleci/openjdk:8-jdk-browsers # ...with this image as the primary container; this is where all `steps` will run

    steps: # a collection of executable commands

      - checkout: # check out source code to working directory
          path: ~/demo

      - restore_cache: # restore the saved cache after the first run or if `pom.xml` has changed
          key: circleci-java-junit-calc-demo # circleci-java-junit-calc-demo-{{ checksum "pom.xml" }}

      - run: mvn dependency:go-offline # gets the project dependencies

      - run: mvn test # run the actual tests

      - save_cache: # saves the project dependencies
          paths:
            - ~/.m2
          key: circleci-java-junit-calc-demo # circleci-java-junit-calc-demo-{{ checksum "pom.xml" }}

      - store_test_results: # uploads the test metadata from the `target/surefire-reports` directory so that it can show up in the CircleCI dashboard.
          path: target/surefire-reports

      - run:
          name: Get API token
          command: |
            echo export token=$(curl -H "Content-Type: application/json" -X POST --data "{ \"client_id\": \"$client_id\",\"client_secret\": \"$client_secret\" }" https://xray.cloud.getxray.app/api/v2/authenticate| tr -d '"') >> $BASH_ENV
            source $BASH_ENV

      - run: 'curl -H "Content-Type: text/xml" -H "Authorization: Bearer $token" --data @target/surefire-reports/TEST-com.xpand.java.CalcTest.xml  "https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=SP"'


In order to submit those results, we'll just need to invoke the REST API (as detailed in Import Execution Results - REST).

However, we do not want to have the Xray API credentials hardcoded in CircleCI's configuration file. Therefore, we'll use some environment variables defined in project settings, including:

  • client_id: the client_id associated with the API key created in the Xray cloud instance 
  • client_secret: the client_secret associated with the API key created in the Xray cloud instance 


Please note

The user associated with Xray's API key must have permission to Create Test and Test Execution Issues.



In.circleci/config.yml a "step" must be included that will use "curl" in order to first obtain a token and then finally submit the results to the REST API, using that token.


export token=$(curl -H "Content-Type: application/json" -X POST --data "{ \"client_id\": \"$client_id\",\"client_secret\": \"$client_secret\" }" https://xray.cloud.getxray.app/api/v2/authenticate| tr -d '"')
curl -H "Content-Type: text/xml" -H "Authorization: Bearer $token" --data @target/surefire-reports/TEST-com.xpand.java.CalcTest.xml  "https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=SP"


We're using "curl" utility that comes in Unix based OS'es but you can easily use another tool to make the HTTP request; however, "curl" is provided in the container used by CircleCI.


Robot Framework example

In this scenario, we want to get visibility of the automated test results from some UI tests implemented in Robot Framework (Python) together with Selenium (using the "robotframework-seleniumlibrary"), and using Chrome for testing.

We need to set up a Git repository containing the code along with the configuration for CircleCI build process.


The tests are implemented in Robot Framework .robot files as follows.

valid_login.robot
*** Settings ***
Documentation     A test suite with a single test for valid login.
...
...               This test has a workflow that is created using keywords in
...               the imported resource file.
Resource          resource.robot

*** Test Cases ***
Valid Login
    [Tags]  UI
    Open Browser To Login Page
    Input Username    demo
    Input Password    mode
    Submit Credentials
    Welcome Page Should Be Open
    [Teardown]    Close Browser 


The CircleCI configuration file .circleci/config.yml contains the definition of the build steps, including running the automated tests and submitting the results.

.circleci/config.yml
# Use the latest 2.1 version of CircleCI pipeline process engine.
# See: https://circleci.com/docs/configuration-reference

# For a detailed guide to building and testing with Python, read the docs:
# https://circleci.com/docs/language-python/ for more details
version: 2.1

# Orbs are reusable packages of CircleCI configuration that you may share across projects, enabling you to create encapsulated, parameterized commands, jobs, and executors that can be used across multiple projects.
# See: https://circleci.com/docs/orb-intro/
orbs:
  # See the Python orb documentation here: https://circleci.com/developer/orbs/orb/circleci/python
  python: circleci/python@2.1.1
  browser-tools: circleci/browser-tools@1.4.6
  
# Define a job to be invoked later in a workflow.
# See: https://circleci.com/docs/jobs-steps/#jobs-overview & https://circleci.com/docs/configuration-reference/#jobs
jobs:
  build-and-test:
    # Specify the execution environment. You can specify an image from Docker Hub or use one of our convenience images from CircleCI's Developer Hub.
    # See: https://circleci.com/docs/executor-intro/ & https://circleci.com/docs/configuration-reference/#executor-job
    docker:
      # Specify the version you desire here
      # See:https://circleci.com/developer/images/image/cimg/python
      - image: cimg/python:3.12-browsers

    # Add steps to the job
    # See: https://circleci.com/docs/jobs-steps/#steps-overview & https://circleci.com/docs/configuration-reference/#steps
    steps:
      # Checkout the code as the first step.
      - checkout
      - python/install-packages:
          pkg-manager: pip
          # app-dir: ~/project/package-directory/  # If your requirements.txt isn't in the root directory.
          # pip-dependency-file: test-requirements.txt  # if you have a different name for your requirements file, maybe one that combines your runtime and test requirements.
      # get server up and running in the background
      - run:
          name: Run webserver to be target by tests
          command: python demoapp/server.py
          background: true
      - run:
          name: Run tests
          # This assumes Robot Framework is installed via the install-package step above
          command: robot -x junit.xml -o output.xml login_tests || true
      - run:
          name: Upload results to Xray DC
          command: |
            echo uploading RF output.xml, if available, to Xray...
            [ -f output.xml ] && curl -H "Content-Type: multipart/form-data" -u $XRAY_USERNAME:$XRAY_PASSWORD -F "file=@output.xml" "$XRAY_SERVER_URL/rest/raven/2.0/import/execution/robot?projectKey=$PROJECT_KEY"
      - store_test_results:
          path: junit.xml
          when: always

# Orchestrate jobs using workflows
# See: https://circleci.com/docs/workflows/ & https://circleci.com/docs/configuration-reference/#workflows
workflows:
  sample: # This is the name of the workflow, feel free to change it to better match your workflow.
    # Inside the workflow, you define the jobs you want to run.
    jobs:
      - build-and-test


In order to submit those results, we'll just need to invoke the REST API (as detailed in Import Execution Results - REST).

However, we do not want to have the Xray API credentials hardcoded in the CircleCI's configuration file. Therefore, we'll use environment variables defined in the project settings, including:

  • client_id: the client_id associated with the API key created in the Xray cloud instance 
  • client_secret: the client_secret associated with the API key created in the Xray cloud instance
  • project_key: the Jira project key


Please note

The user associated with the Xray's API key must have permissions to Create Test and Test Execution Issues.



In .circleci/config.yml a "step" must be included that will use "curl" in order to first obtain a token and then finally submit the results to the REST API, using that token.


export token=$(curl -H "Content-Type: application/json" -X POST --data "{ \"client_id\": \"$client_id\",\"client_secret\": \"$client_secret\" }" https://xray.cloud.getxray.app/api/v2/authenticate| tr -d '"')
curl -H "Content-Type: text/xml" -H "Authorization: Bearer $token" --data @"output.xml" "https://xray.cloud.getxray.app/api/v2/import/execution/robot?projectKey=$project_key"


We're using "curl" utility that comes in Unix based OS'es but you can easily use another tool to make the HTTP request.


 


 


References