Overview

RJT allows users to configure and invoke remote jobs in different CI/CD tools without leaving Xray improving tester performance and streamlining the workflow. 

Most pipelines are triggered by a commit action but sometimes we have the necessity to trigger a remote job to perform some actions, such as:

• Validate a change in a specific feature
• Validate a new deployment or new environment
• Validate new tests on the fly
• Run automation from Xray Test Plan or Test Execution

The remote job can perform all sort of tasks, including building, deploying the project to an environment, and/or running automated tests.

Most common use is to trigger the execution of automated tests.

In this example we are configuring a Remote Job Trigger for Jenkins that execute Playwright tests and send the execution results back to Xray.

Prerequisites

Configure a new RJT for Jenkins in Xray

This example requires configuration in both sides (Xray and Jenkins) so that we can take most advantage of the combination of both tools.

The jenkinsfile will configure a multi-step pipeline that will extract the Playwright test code, execute it and ship the execution results back to Xray.

Configure Jenkins using a jenkinsfile

We use a jenkinsfile to configure the pipeline in Jenkins.

pipeline {
  parameters {
    string(name: 'projectKey', defaultValue: '')
    string(name: 'testPlanKey', defaultValue: '')
  }
  agent {
    docker { 
      image 'mcr.microsoft.com/playwright:v1.27.0-focal'
    } 
  }
  stages {
    stage('install playwright') {
      steps {
        sh '''
          npm i -D @playwright/test
          npx playwright install
        '''
      }
    }
    stage('test') {
      steps {
        catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
          sh '''
            PLAYWRIGHT_JUNIT_OUTPUT_NAME=xray-report.xml npx playwright test
          '''
        }
      }
    }
    stage('Import results to Xray') {
      steps {
        step([$class: 'XrayImportBuilder', endpointName: '/junit', importFilePath: 'xray-report.xml', importToSameExecution: 'true', projectKey: params.projectKey, testPlanKey: params.testPlanKey, fixVersion: '1.2', revision: '131', serverInstance: '10be58cc-2776-49a7-be60-b615dc99f4c0'])
      }
    }
    stage('Extract Variable from log'){
      steps {
        script {
          def logContent = Jenkins.getInstance().getItemByFullName(env.JOB_NAME).getBuildByNumber(Integer.parseInt(env.BUILD_NUMBER)).logFile.text
          env.testExecs = (logContent =~ /XRAY_TEST_EXECS:.*/).findAll().first()
          echo testExecs
         }
      }
    }
  }
  post 
  {
    always {
      junit '*.xml'
    }
  }
}

On the above jenkinsfile we are defining two parameters that will be passed when the build is invoked.

...
parameters {
    string(name: 'projectKey', defaultValue: '')
    string(name: 'testPlanKey', defaultValue: '')
  }
...

The parameters received can be used in the remaining steps of the pipeline. In order to define what are the parameters we are adding to the pipeline we have added a parameters section with he name of the parameter and a possible default value.

Within the step to run the tests we have added a function to make sure that the next step is performed even if the actual step will fail. This is important because if the tests fail we must assure that the results are sent to Xray.

...
catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
...

Using the catchError method we are forcing the step

Once the jenkinsfile is configure in Jenkins it is ready to start to perform the stages and steps defined within it.

We are going to dive into the last part of the jenkinsfile, where we send the results back to Xray, in another section. 

Configure a Remote Jobs Trigger in Xray for Jenkins

Notes:

• By default it will execute tests for the 3 browser types available (that is why we are forcing it to execute for only one browser)

Integrating with Xray

As we saw in the above example, where we are producing Junit reports with the result of the tests, it is now a matter of importing those results to your Jira instance. You can do this by simply submitting automation results to Xray through the REST API, by using one of the available CI/CD plugins (e.g. for Jenkins) or using the Jira interface to do so.

curl -H "Content-Type: multipart/form-data" -u admin:admin -F "file=@junit.xml" 'http://<LOCAL_JIRA_INSTANCE>/rest/raven/1.0/import/execution/junit?projectKey=COM&testPlanKey=COM-9'

Tips

• after results are imported in Jira, Tests can be linked to existing requirements/user stories, so you can track the impact of their coverage.
• results from multiple builds can be linked to an existing Test Plan in order to facilitate the analysis of test result trends across builds.
• results can be associated with a Test Environment, in case you want to analyze coverage and test results by that environment later on. A Test Environment can be a testing stage (e.g. dev, staging, preprod, prod) or an identifier of the device/application used to interact with the system (e.g. browser, mobile OS).

References

• https://playwright.dev/docs/test-intro/
• https://playwright.dev/