Page History
Versions Compared
Key
- This line was added.
- This line was removed.
- Formatting was changed.
Info | ||
---|---|---|
| ||
|
Note | ||||||
---|---|---|---|---|---|---|
| ||||||
|
Overview
|
Image Modified
RJT allows users to configure and invoke remote jobs in different CI/CD tools without leaving Xray improving tester performance and streamlining the workflow.
Most pipelines are triggered by a commit action but sometimes we have the necessity to trigger a remote job to perform some actions, such as:
- Validate a change in a specific feature
- Validate a new deployment or new environment
- Validate new tests on the fly
- Run automation from Xray Test Plan or Test Execution
The remote job can perform all sort of tasks, including building, deploying the project to an environment, and/or running automated tests.
Most common use is to trigger the execution of automated tests.
In this example we are configuring a Remote Job Trigger for Jenkins that execute Playwright tests and send the execution results back to Xray.
Prerequisites
Expand |
---|
For this example we will use Jenkins as the CI/CD tool that will execute Playwright tests. What you need:
|
Configure a new RJT for Jenkins in Xray
This example requires configuration in both sides (Xray and Jenkins) so that we can take most advantage of the combination of both tools.
The jenkinsfile will configure a multi-step pipeline that will extract the Playwright test code, execute it and ship the execution results back to Xray.
Configure Jenkins using a jenkinsfile
Configure a Remote Jobs Trigger in Xray for JenkinsWe use a jenkinsfile to configure the pipeline in Jenkins.
Code Block | |||||||
---|---|---|---|---|---|---|---|
| |||||||
pipeline { parameters { string(name: 'projectKey', defaultValue: '')const config = require ("../config.json"); // models/Login.js class LoginPage { constructor(page) { string(name: this.page = page;'testPlanKey', defaultValue: '') } agent } { asyncdocker navigate(){ { await this.page.goto(config.endpoint);image 'mcr.microsoft.com/playwright:v1.27.0-focal' } } stages { async login(username, passwordstage('install playwright') { await this.page.fill(config.username_field, username);steps { await this.page.fill(config.password_field, password);sh ''' await this.page.click(config.login_button); } npm i -D @playwright/test async getInnerText(){ npx playwright install return this.page.innerText("p"); } }''' module.exports = { LoginPage }; |
plus a configuration file where we have the identifiers that will match the elements in the page
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
{ "endpoint" : "https://robotwebdemo.onrender.com/",} "login_button" : "id=login_button",stage('test') { "password_field" :"input[id=\"password_field\"]", "username_field" : "input[id=\"username_field\"]" } |
And define the test that will assert if the operation is successful or not
Code Block | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
import {it, describe, expect} from "@playwright/test" import { LoginPage } from "./models/Login"; describe("Login validations", () => { it('Login with valid credentials', async({page}) => steps { catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') { const loginPage = new LoginPage(page);sh ''' await loginPage.navigate(); PLAYWRIGHT_JUNIT_OUTPUT_NAME=xray-report.xml npx await loginPage.login("demo","mode");playwright test const name =''' await loginPage.getInnerText(); } expect(name).toBe('Login succeeded. Now you can logout.'); } }); itstage('LoginImport withresults invalidto credentials', async({page}Xray') => { const loginPage = new LoginPage(page);steps { await loginPage.navigate(); await loginPage.login("demo","mode1"); const name = await loginPage.getInnerText(); expect(name).toBe('Login failed. Invalid user name and/or password.'); }); }) |
The Playwright Test Runner provides a Jest like way of describing test scenarios, here you can see that it uses 'it, describe, expect'.
These are simple tests that will validate the login functionality by accessing the demo site, inserting the username and password (in one test with valid credentials and in another with invalid credentials), clicking the login button and validating if the page returned is the one that matches your expectation.
For the below example we will do a small change to force a failure, so in the login.spec.ts file remove "/or" from the expectation on the Test ' Login with invalid credentials', this is the end result:
Code Block | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
import { test, expect } from "@playwright/test" import { LoginPage } from "./models/Login"; test.describe("Login validations", () => { test('Login with valid credentials', async({ page }) => step([$class: 'XrayImportBuilder', endpointName: '/junit', importFilePath: 'xray-report.xml', importToSameExecution: 'true', projectKey: params.projectKey, testPlanKey: params.testPlanKey, fixVersion: '1.2', revision: '131', serverInstance: '10be58cc-2776-49a7-be60-b615dc99f4c0']) } } stage('Extract Variable from log'){ const loginPage = new LoginPage(page);steps { await loginPage.navigate(); await loginPage.login("demo","mode"); script { constdef namelogContent = await loginPageJenkins.getInnerTextgetInstance(); expect(name).toBe('Login succeeded. Now you can logout.');.getItemByFullName(env.JOB_NAME).getBuildByNumber(Integer.parseInt(env.BUILD_NUMBER)).logFile.text }); test('Login with invalid credentials', async({ page }) => {env.testExecs = (logContent =~ /XRAY_TEST_EXECS:.*/).findAll().first() const loginPage = new LoginPage(page);echo testExecs await loginPage.navigate(); } } await loginPage.login("demo","mode1"); } } post { const name = await loginPage.getInnerText();always { expect(name).toBe('Login failed. Invalid user name and password.'); }); }) |
Once the code is implemented (and we will make it fail on purpose on the 'Login with invalid credentials' test due to missing word, to show the failure reports), can be executed with the following command:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
npx folio -p browserName=chromium --reporter=junit,line --test-match=login.spec.ts |
First, define one extra parameter: "browserName" in order to execute the tests only with the chrome browser (chromium), otherwise the default behaviour is to execute the tests for the three available browsers (chromium, firefox and webkit).
The results are immediately available in the terminal
Image Removed
In this example, one test has failed and the other one has succeed, the output generated in the terminal is the above one and the corresponding Junit report is below:
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
<testsuites id="" name="" tests="2" failures="1" skipped="0" errors="0" time="2.592">
<testsuite name="login.spec.ts" timestamp="1617094735952" hostname="" tests="2" failures="1" skipped="0" time="2.37" errors="0">
<testcase name="Login validations Login with valid credentials" classname="login.spec.ts Login validations" time="1.358">
</testcase>
<testcase name="Login validations Login with invalid credentials" classname="login.spec.ts Login validations" time="1.012">
<failure message="login.spec.ts:14:5 Login with invalid credentials" type="FAILURE">
login.spec.ts:14:5 › Login validations Login with invalid credentials ============================
browserName=webkit, headful=false, slowMo=0, video=false, screenshotOnFailure=false
Error: expect(received).toBe(expected) // Object.is equality
Expected: "Login failed. Invalid user name and password."
Received: "Login failed. Invalid user name and/or password."
17 | await loginPage.login("demo","mode1");
18 | const name = await loginPage.getInnerText();
> 19 | expect(name).toBe('Login failed. Invalid user name and password.');
| ^
20 | });
21 | })
at /Users/cristianocunha/Documents/Projects/Playwrighttest/login.spec.ts:19:22
at runNextTicks (internal/process/task_queues.js:58:5)
at processImmediate (internal/timers.js:434:9)
at WorkerRunner._runTestWithFixturesAndHooks (/Users/cristianocunha/Documents/Projects/Playwrighttest/node_modules/folio/out/workerRunner.js:198:17)
</failure>
</testcase>
</testsuite>
</testsuites> |
junit '*.xml'
}
}
} |
On the above jenkinsfile we are defining two parameters that will be passed when the build is invoked.
Code Block | ||||
---|---|---|---|---|
| ||||
...
parameters {
string(name: 'projectKey', defaultValue: '')
string(name: 'testPlanKey', defaultValue: '')
}
... |
The parameters received can be used in the remaining steps of the pipeline. In order to define what are the parameters we are adding to the pipeline we have added a parameters section with he name of the parameter and a possible default value.
Configure a Remote Jobs Trigger in Xray for Jenkins
Notes:
- By default it will execute tests for the 3 browser types available (that is why we are forcing it to execute for only one browser)
- By default all the tests will be executed in headless mode
- Folio command line will search and execute all tests in the format: "**/?(*.)+(spec|test).[jt]s" In order to get the Junit test report please follow this section.
Integrating with Xray
As we saw in the above example, where we are producing Junit reports with the result of the tests, it is now a matter of importing those results to your Jira instance. You can do this by simply submitting automation results to Xray through the REST API, by using one of the available CI/CD plugins (e.g. for Jenkins) or using the Jira interface to do so.
UI Tabs | ||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Tips
- after results are imported in Jira, Tests can be linked to existing requirements/user stories, so you can track the impact of their coverage.
- results from multiple builds can be linked to an existing Test Plan in order to facilitate the analysis of test result trends across builds.
- results can be associated with a Test Environment, in case you want to analyze coverage and test results by that environment later on. A Test Environment can be a testing stage (e.g. dev, staging, preprod, prod) or an identifier of the device/application used to interact with the system (e.g. browser, mobile OS).
References
Table of Contents | ||
---|---|---|
|
CSS Stylesheet |
---|
.toc-btf { position: fixed; } |
Table of Contents | ||
---|---|---|
|
CSS Stylesheet |
---|
.toc-btf { position: fixed; } |
UI Tab | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||
Jenkinsfile
|