Page History
Versions Compared
Key
- This line was added.
- This line was removed.
- Formatting was changed.
Image Added
Info | ||
---|---|---|
| ||
|
Note | ||||||
---|---|---|---|---|---|---|
| ||||||
|
Overview
Image Removed
RJT allows users to configure and invoke remote jobs in different CI/CD tools without leaving Xray improving tester performance and streamlining the workflow.
Most pipelines are triggered by a commit action but sometimes we have the necessity to trigger a remote job to perform some actions, such as:
Validate a change in a specific featureThe remote job can perform all sort of tasks, including building, deploying the project to an environment, and/or running automated tests.
Most common use is to trigger the execution of automated tests.
In this example we are configuring a Remote Job Trigger for Jenkins that execute executes Playwright tests and send sends the execution results back to Xray.
Prerequisites
Expand |
---|
For this example we will use Jenkins as the CI/CD tool that will execute Playwright tests. What you need:
|
Configure a new RJT for Jenkins in Xray
This example requires configuration in both sides (Xray and Jenkins) so that we can take most advantage take advantage of the combination of both tools.
The jenkinsfile will configure a multi-step pipeline that extracts the Playwright test code, execute executes it and ship ships the execution results back to Xray.
Configure Jenkins using a jenkinsfile
We use a jenkinsfile to configure the pipeline in Jenkins.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
pipeline { parameters { string(name: 'projectKey', defaultValue: '') string(name: 'testPlanKey', defaultValue: '') } agent { docker { image 'mcr.microsoft.com/playwright:v1.27.0-focal' } } stages { stage('install playwright') { steps { sh ''' npm i -D @playwright/test npx playwright install ''' } } stage('test') { steps { catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') { sh ''' PLAYWRIGHT_JUNIT_OUTPUT_NAME=xray-report.xml npx playwright test ''' } } } stage('Import results to Xray') { steps { step([$class: 'XrayImportBuilder', endpointName: '/junit', importFilePath: 'xray-report.xml', importToSameExecution: 'true', projectKey: params.projectKey, testPlanKey: params.testPlanKey, fixVersion: '1.2', revision: '131', serverInstance: '10be58cc-2776-49a7-be60-b615dc99f4c0']) } } stage('Extract Variable from log'){ steps { script { def logContent = Jenkins.getInstance().getItemByFullName(env.JOB_NAME).getBuildByNumber(Integer.parseInt(env.BUILD_NUMBER)).logFile.text env.testExecs = (logContent =~ /XRAY_TEST_EXECS:.*/).findAll().first() echo testExecs } } } } post { always { junit '*.xml' } } } |
On the above jenkinsfile we are defining two parameters that will be passed when the build is invoked.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
... parameters { string(name: 'projectKey', defaultValue: '') string(name: 'testPlanKey', defaultValue: '') } ... |
The parameters received can received parameters can be used in the remaining steps of the pipeline. In order to define what are the parameters we have added a parameters section with he name of the parameter and a possible default value.
Within the step to run the tests we have added a function to make sure that the next step is performed even if the actual step will failfails. This is important to assure that the results are sent to Xray.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
... catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') { ... |
Using the catchError
method in case of an error, we are forcing the step to continue the Pipeline execution from the statement following the catchError
step.
The behavior of the step when an exception is thrown can be configured to print a message, set a build result other than failure, change the stage result, or ignore certain kinds of exceptions that are used to interrupt the build.
Once the jenkinsfile is uploaded in Jenkins it is ready to start to perform the tasks defined in the stages and steps.
We are going to dive into the last part of the jenkinsfile, where we send the results back to Xray, in another section.
Configure a Remote Jobs Trigger in Xray for Jenkins
In order to use the configuration of a RJT in Xray you need to access the Project Settings
area and click on the Remote Jobs Trigger
option.
This will open the Remote Jobs Trigger
configuration page where we can configure remote jobs for Jenkins, Bamboo, GitLab, GitHub and Azure DevOps. You can activate and deactivate the configurations by switching the toggles next to each.
Info | ||
---|---|---|
| ||
Only one configuration can be active at each time. |
Once you have activated the configuration that you are interested, in our case Jenkins Configuration, click the New Jenkins Configuration
to configure your new job. This opens a new form that must filled with proper information.
Fill the Configuration Name
and the Job Name
with meaningful information that allows you to know what is the purpose of the job just by reading the name.
The Job Name
is the name of the project/job in Jenkins, make sure that they match.
Next we have the API Token
, this can be defined in the Pipeline project configuration in Jenkins. Access your project page in Jenkins and scroll down to the Build Triggers
area.
The Authentication Token
defined in Jenkins must be the same defined in Xray API Token
of the Remote Jobs Trigger
configuration, in our case 'MyToken'.
In the Username
field define the name of the Jenkins user with proper permissions to invoke the Jenkins Job. Make sure it has the right permissions in Jenkins or the job will return an authentication error when used.
Fill the API URL
with the endpoint base URL of your Jenkins instance and in the Password
field put the password used by the user identified by the Username above.
Finally we have reached the last configuration fields available: Parameters (Optional), that is a list of key/value pairs that you have defined in you Jenkins pipeline and want to pass from Xray.
You can define static ones or you can use dynamic filled fields available in Xray (more info here). In our case we want to pass the Project key and the Test Plan key so that we can use them when shipping the execution results back to Xray.
For that we use the following options available from Xray:
- ${PROJECT_KEY} that will be filled with the key of the project from where the job is called.
- ${ISSUE_KEY} that will be filled with the issue key, in our case the Test Plan key, from where the job was started from.
Info | ||
---|---|---|
| ||
Make sure that the names of the parameters match between what you have configured in the jenkinsfile and the Remote Jobs Triggers configuration. |
The configuration with all the fields filled will look like this:
Once done click the Save
button and activate the configuration by switching the Status
toggle to on:
Integrating with Xray
As we saw in the above example, where we are producing Junit reports with the result of the tests, it is now a matter of importing those results to your Jira instance. You can do this by simply submitting automation results to Xray through the REST API, by using one of the available CI/CD plugins (e.g. for Jenkins) or using the Jira interface to do so.
ui-tabsTrigger
Once the configuration is done and active you can navigate to, e. g. the Test Plan, from where you want to trigger the job, there you will see that a new button is available to trigger the job you just have configured.
Image Added
When you choose to call the Remote Jobs Trigger
job a new confirmation window will appear with the status of the invocation, either success or failure.
Image Added
Info |
---|
Notice that the status is of the invocation of the job only (not the status of the execution of the job). If you get a failure status please review the configuration in the previous sections. |
Send results back to Xray
In the above example we have defined a job in another tool and invoked it from Xray. This job will execute the Playwright tests and generate a Junit report.
Now we need to have those results back into Xray for full visibility. The following section will show you two ways you can use to have those results shipped back into Xray.
UI Tab | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
| API |
Code Block | ||||
---|---|---|---|---|
| ||||
curl -H "Content-Type: multipart/form-data" -u admin:admin -F "file=@junit.xml" 'http://<LOCAL_JIRA_INSTANCE>/rest/raven/1.0/import/execution/junit?projectKey=COM&testPlanKey=COM-9' |
With this command, you will create a new Test Execution in the referred Test Plan with a generic summary and two tests with a summary based on the test name.
Image Removed
On Xray, you can see the tests and you can identify which tests are failing or passing. Below you can see two tests (for valid and invalid credentials):
Image Removed
You can also notice that the summary is now defined based on the files we used for uploading the test results.
title | Jenkinsfile |
---|
Jenkinsfile
title | Jenkins |
---|
Jenkins
As you can see below we are adding a post-build action using the "Xray: Results Import Task" (from the Xray plugin available), where we have some options. For now, we will focus on two of those, one called "Junit XML" (simpler) and another called "Junit XML multipart" (both are explained below and will require two extra files).
Junit XML
- the Jira instance (where you have your Xray instance installed)
- the format as "JUnit XML"
- the test results file we want to import
- the Project key corresponding of the project in Jira where the results will be imported
Image Removed
Tests implemented using Jest will have a corresponding Test entity in Xray. Once results are uploaded, Test issues corresponding to the Jest tests are auto-provisioned, unless they already exist.
Image Removed
Xray uses a concatenation of the suite name and the test name as the the unique identifier for the test.
In Xray, results are stored in a Test Execution, usually a new one. The Test Execution contains a Test Run per each test that was executed using playwright-test runner.
Image Removed
Detailed results, including logs and exceptions reported during the execution of the test, can be seen on the execution screen details of each Test Run, accessible through the Execution details:
Image Removed
As you can see here:
Image Removed
Tips
- after results are imported in Jira, Tests can be linked to existing requirements/user stories, so you can track the impact of their coverage.
- results from multiple builds can be linked to an existing Test Plan in order to facilitate the analysis of test result trends across builds.
- results can be associated with a Test Environment, in case you want to analyze coverage and test results by that environment later on. A Test Environment can be a testing stage (e.g. dev, staging, preprod, prod) or an identifier of the device/application used to interact with the system (e.g. browser, mobile OS).
References
| |||||||||||||||||||
StepsThe Xray plugin for Jenkins provides steps that enable you to import the results to Xray.
Let us look into more detail over the step Remember that in our jenkinsfile we have defined parameters in the beginning of the file and one step to import the results into Xray.
With these options we are saying to the Pipeline that we are going to receive two parameters:
On the import step we are using those parameters to define what is the Jira Project that we are import into and what is the Test Plan we will associate the result to. Let's look into the step in more detail, explaining each option:
Once the pipeline ends with success you can check details in the console output of the build in Jenkins to make sure all went as expected. Image Added We can see that it has imported the execution results with success and that it has created a new Test Execution: When accessing the Test Plan we can see that the results were ingested and the overall status is visible from the detail view of the issue. Image Added The details of the results are present in the details of the Test Execution, after clicking in the Test Execution link, we need to click the Execution Details option in the drop down menu: Image Added This opens the detail page of the Test Execution where we have the details sent through the import of the test results: Image Added |
Tips
- Make sure that you define parameters in the jenkinsfile if you want to pass some.
- When defining parameters make sure that the parameters names match in both tools.
- The Authentication Token from Jenkins pipeline must match the API Token from Xray side.
- Confirm that the user that you are using have the correct permissions in both tools.
References
- Triggering Remote Jobs
- Jenkins pipeline integration
- Integration with Jenkins
- Playwright
- https://playwright.dev/docs/test-intro/ https://playwright.dev/
Table of Contents | ||
---|---|---|
|
CSS Stylesheet |
---|
.toc-btf { position: fixed; } |
Table of Contents | ||
---|---|---|
|
CSS Stylesheet |
---|
.toc-btf { position: fixed; } |