CircleCI is a well-known CI/CD tool available on-premises and as SaaS.
Xray does not provide yet a plugin for CircleCI. However, it is easy to setup CircleCI in order to integrate it with Xray.
Since Xray provides a full REST API, you may interact with Xray, for submitting results for example.
JUnit example
In this scenario, we want to get visibility of the automated test results from some tests implemented in Java, using the JUnit framework.
This recipe could also be applied for other frameworks such as NUnit, TestNG or Robot.
We need to setup a project based on a Git repository containing the code along with the configuration for CircleCI build process.
The tests are implemented in a JUnit class as follows.
The CircleCI configuration file .circleci/config.ym
l contains the definition of the build steps, including running the automated tests and submitting the results.
version: 2 # use CircleCI 2.0 jobs: # a collection of steps build: # runs not using Workflows must have a `build` job as entry point working_directory: ~/demo/java-junit-calc # directory where steps will run docker: # run the steps with Docker - image: circleci/openjdk:8-jdk-browsers # ...with this image as the primary container; this is where all `steps` will run steps: # a collection of executable commands - checkout: # check out source code to working directory path: ~/demo - restore_cache: # restore the saved cache after the first run or if `pom.xml` has changed key: circleci-java-junit-calc-demo # circleci-java-junit-calc-demo-{{ checksum "pom.xml" }} - run: mvn dependency:go-offline # gets the project dependencies - run: mvn test # run the actual tests - save_cache: # saves the project dependencies paths: - ~/.m2 key: circleci-java-junit-calc-demo # circleci-java-junit-calc-demo-{{ checksum "pom.xml" }} - store_test_results: # uploads the test metadata from the `target/surefire-reports` directory so that it can show up in the CircleCI dashboard. path: target/surefire-reports - run: name: Get API token command: | echo export token=$(curl -H "Content-Type: application/json" -X POST --data "{ \"client_id\": \"$client_id\",\"client_secret\": \"$client_secret\" }" https://xray.cloud.getxray.app/api/v2/authenticate| tr -d '"') >> $BASH_ENV source $BASH_ENV - run: 'curl -H "Content-Type: text/xml" -H "Authorization: Bearer $token" --data @target/surefire-reports/TEST-com.xpand.java.CalcTest.xml "https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=SP"'
In order to submit those results, we'll just need to invoke the REST API (as detailed in Import Execution Results - REST).
However, we do not want to have the Xray API credentials hardcoded in CircleCI's configuration file. Therefore, we'll use some environment variables defined in project settings, including:
- client_id: the client_id associated with the API key created in the Xray cloud instance
- client_secret: the client_secret associated with the API key created in the Xray cloud instance
Please note
The user associated with Xray's API key must have permission to Create Test and Test Execution Issues.
In.circleci/config.ym
l a "step" must be included that will use "curl" in order to first obtain a token and then finally submit the results to the REST API, using that token.
export token=$(curl -H "Content-Type: application/json" -X POST --data "{ \"client_id\": \"$client_id\",\"client_secret\": \"$client_secret\" }" https://xray.cloud.getxray.app/api/v2/authenticate| tr -d '"')
curl -H "Content-Type: text/xml" -H "Authorization: Bearer $token" --data @target/surefire-reports/TEST-com.xpand.java.CalcTest.xml "https://xray.cloud.getxray.app/api/v2/import/execution/junit?projectKey=SP"
We're using "curl" utility that comes in Unix based OS'es but you can easily use another tool to make the HTTP request; however, "curl" is provided in the container used by CircleCI.
Robot Framework example
In this scenario, we want to get visibility of the automated test results from some UI tests implemented in Robot Framework (Python) together with Selenium (using the "robotframework-seleniumlibrary"), and using Chrome for testing.
We need to set up a Git repository containing the code along with the configuration for CircleCI build process.
The tests are implemented in Robot Framework .robot files as follows.
The CircleCI configuration file .circleci/config.yml
contains the definition of the build steps, including running the automated tests and submitting the results.
# Use the latest 2.1 version of CircleCI pipeline process engine. # See: https://circleci.com/docs/configuration-reference # For a detailed guide to building and testing with Python, read the docs: # https://circleci.com/docs/language-python/ for more details version: 2.1 # Orbs are reusable packages of CircleCI configuration that you may share across projects, enabling you to create encapsulated, parameterized commands, jobs, and executors that can be used across multiple projects. # See: https://circleci.com/docs/orb-intro/ orbs: # See the Python orb documentation here: https://circleci.com/developer/orbs/orb/circleci/python python: circleci/python@2.1.1 browser-tools: circleci/browser-tools@1.4.6 # Define a job to be invoked later in a workflow. # See: https://circleci.com/docs/jobs-steps/#jobs-overview & https://circleci.com/docs/configuration-reference/#jobs jobs: build-and-test: # Specify the execution environment. You can specify an image from Docker Hub or use one of our convenience images from CircleCI's Developer Hub. # See: https://circleci.com/docs/executor-intro/ & https://circleci.com/docs/configuration-reference/#executor-job docker: # Specify the version you desire here # See:https://circleci.com/developer/images/image/cimg/python - image: cimg/python:3.12-browsers # Add steps to the job # See: https://circleci.com/docs/jobs-steps/#steps-overview & https://circleci.com/docs/configuration-reference/#steps steps: # Checkout the code as the first step. - checkout - python/install-packages: pkg-manager: pip # app-dir: ~/project/package-directory/ # If your requirements.txt isn't in the root directory. # pip-dependency-file: test-requirements.txt # if you have a different name for your requirements file, maybe one that combines your runtime and test requirements. # get server up and running in the background - run: name: Run webserver to be target by tests command: python demoapp/server.py background: true - run: name: Run tests # This assumes Robot Framework is installed via the install-package step above command: robot -x junit.xml -o output.xml login_tests || true - run: name: Upload results to Xray DC command: | echo uploading RF output.xml, if available, to Xray... [ -f output.xml ] && curl -H "Content-Type: multipart/form-data" -u $XRAY_USERNAME:$XRAY_PASSWORD -F "file=@output.xml" "$XRAY_SERVER_URL/rest/raven/2.0/import/execution/robot?projectKey=$PROJECT_KEY" - store_test_results: path: junit.xml when: always # Orchestrate jobs using workflows # See: https://circleci.com/docs/workflows/ & https://circleci.com/docs/configuration-reference/#workflows workflows: sample: # This is the name of the workflow, feel free to change it to better match your workflow. # Inside the workflow, you define the jobs you want to run. jobs: - build-and-test
In order to submit those results, we'll just need to invoke the REST API (as detailed in Import Execution Results - REST).
However, we do not want to have the Xray API credentials hardcoded in the CircleCI's configuration file. Therefore, we'll use environment variables defined in the project settings, including:
- client_id: the client_id associated with the API key created in the Xray cloud instance
- client_secret: the client_secret associated with the API key created in the Xray cloud instance
- project_key: the Jira project key
Please note
The user associated with the Xray's API key must have permissions to Create Test and Test Execution Issues.
In .circleci/config.yml
a "step" must be included that will use "curl" in order to first obtain a token and then finally submit the results to the REST API, using that token.
export token=$(curl -H "Content-Type: application/json" -X POST --data "{ \"client_id\": \"$client_id\",\"client_secret\": \"$client_secret\" }" https://xray.cloud.getxray.app/api/v2/authenticate| tr -d '"')
curl -H "Content-Type: text/xml" -H "Authorization: Bearer $token" --data @"output.xml" "https://xray.cloud.getxray.app/api/v2/import/execution/robot?projectKey=$project_key"
We're using "curl" utility that comes in Unix based OS'es but you can easily use another tool to make the HTTP request.