...

In this tutorial, we will create some UI tests as Cucumber Scenario(s)/Scenario Outline(s) and use Cypress to implement the tests in JavaScript.

Requirements

• nodejs
• npm packages
  • cypress
  • cypress-cucumber-preprocessor
  • cucumber-json-merge

...

For the purpose of this tutorial, we 'll will use a dummy website (source-code here) containing just a few pages to support login/logout kind of features; we aim to test precisely those features.

...

We need to configure Cypress to use the cypress-cucumber-preprocessor, which provides the ability of understanding to understand .feature files and also of producing to produce Cucumber JSON reports.

const cucumber = require('cypress-cucumber-preprocessor').default

/**
 * @type {Cypress.PluginConfig}
 */
module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config
  on('file:preprocessor', cucumber())
}

...

In Cypress' main configuration file, we define the base URL of the website under test, the regex of the files that contain the test scenarios (i.e. <...>.feature files). Other options may be defined  (defined e.g for bypassing chromeWebSecurity, additional reporters, the ability to upload results to Cypress infrastructure in the cloud, etc).

...

{
  "baseUrl": "https://robotwebdemo.herokuapp.com/",
  "testFiles": "**/*.feature",
  "ignoreTestFiles": [
    "*.js",
    "*.md"
  ],
  "reporter": "junit",
  "reporterOptions": {
    "mochaFile": "test-results/test-output-[hash].xml"
  },
  "chromeWebSecurity": false,
  "projectId": "bfi83g"
}

Next, here is an example of the contents of package.json.

{
  "name": "cypress-cucumber-robotdemo",
  "version": "1.0.0",
  "description": "An example for Cypress and Cucumber usage using Robot login demo website",
  "main": "index.js",
  "scripts": {
    "cypress:open:local": "CYPRESS_ENV=localhost npm run cypress:open",
    "cypress:open:prod": "CYPRESS_ENV=production npm run cypress:open",
    "cypress:open": "cypress open",
    "test:local": "CYPRESS_ENV=localhost npm run test  --spec 'cypress/integration/**/*.feature",
    "test:prod": "CYPRESS_ENV=production npm run test",
    "test": "cypress run --spec 'features/**/*.feature' --config integrationFolder=.",
    "test:debug:local": "CYPRESS_ENV=localhost npm run test:debug",
    "test:debug:prod": "CYPRESS_ENV=production npm run test:debug",
    "test:debug": "cypress run --headed --browser chrome --env TAGS='@e2e-test' --spec 'cypress/integration/**/*.feature'",
    "test:pull-features": "git submodule update --remote gherkin-features && cp -rf gherkin-features/* cypress/integration && node ./scripts/remove-old-features.js",
    "attach_screenshots": "node attach_screenshots.js"
  },
  "author": "",
  "license": "Private",
  "dependencies": {
    "axios": "^0.18.0",
    "cucumber-json-merge": "0.0.4",
    "fs-extra": "^7.0.1",
    "glob": "^7.1.3"
  },
  "devDependencies": {
    "cypress": "^5.5.0",
    "cypress-cucumber-preprocessor": "^4.0.0",
    "eslint": "^5.13.0",
    "eslint-config-airbnb-base": "^12.1.0",
    "eslint-config-prettier": "^2.9.0",
    "eslint-plugin-import": "^2.11.0",
    "eslint-plugin-prettier": "^2.6.0",
    "husky": "^1.3.1",
    "lint-staged": "^8.1.3"
  },
  "cypress-cucumber-preprocessor": {
    "nonGlobalStepDefinitions": true,
    "cucumberJson": {
      "generate": true,
      "outputFolder": "cypress/cucumber-json",
      "filePrefix": "",
      "fileSuffix": ".cucumber"
    }
  },
  "husky": {
    "hooks": {
      "pre-commit": "lint-staged"
    }
  },
  "lint-staged": {
    "*.js": [
      "eslint",
      "git add"
    ]
  }
}

Before moving into the actual implementation, we need to decide which workflow we'll use: do we want to use Xray/Jira as the master for writing the declarative specification (i.e. the Gherkin based Scenarios), or do we want to Before moving into the actual implementation, we need to decide is which workflow we'll use: do we want to use Xray/Jira as the master for writing the declarative specification (i.e. the Gherkin based Scenarios), or do we want to manage those outside using some editor and store them in Git, for example?

...

Using Jira and Xray as master

This section assumes using you will use Xray as master, i.e. the place that you'll be using to edit the specifications (e.g. the scenarios that are part of .feature files).

...

If you have it, then you can just use the "Create Test" on that issue to create the Scenario/Scenario Outline and have it automatically linked back to the Story/"requirement.".

Otherwise, you can create the Test using the standard (issue) Create action from Jira's top menu. 

...

In this case, we'll create a Cucumber Test, of Cucumber Type "Scenario.".

We can fill out the Gherkin statements immediately on the Jira issue "create dialog" or we can create the Test issue first and fill out the details on the next screen, from within the Test issue. In the latter case, we can take advantage of the built-in Gherkin editor which provides auto-complete of for Gherkin steps.

After the Test is created it will impact the coverage of related "requirement," , if any.

The coverage and the test results can be tracked in the "requirement" side (e.g. user story). In this case, you may see that coverage changed from being UNCOVERED to NOTRUN (i.e. covered and with at least one test not run).

...

The related statement's code is managed outside of Jira and stored in Git, for example.

In Cypress, tests related the test code is stored under cypress/integration directory, which itself contains several other directories. In this case, we've organized them as follows:

...

@REQ_CALC-7905
Feature: As a user, I can login the application
	#As a user, I can login the application

	
	@TEST_CALC-7903
	Scenario Outline: Login With Invalid Credentials Should Fail
		Given browser is opened to login page
		When user "<username>" logs in with password "<password>"
		Then error page should be open
		
			Examples: 
				| username | password | 
				| invalid  | mode     | 
				| demo     | invalid  | 
				| invalid  | invalid  | 
				| demo     | mode     |	

	
	@TEST_CALC-7902
	Scenario: Invalid Login
		Given browser is opened to login page
		When user "dummy" logs in with password "password"
		Then error page should be open	

	
	@TEST_CALC-7901
	Scenario: Valid Login
		Given browser is opened to login page
		When user "demo" logs in with password "mode"
		Then welcome page should be open

@REQ_CALC-7906
Feature: As a user, I can logout the application
	#As a user, I can logout the application

	
	@TEST_CALC-7904
	Scenario: Valid Logout
		Given user is on the welcome page
		When user chooses to logout
		Then login page should be open

...

A new Test Execution will be created (unless you originally exported the Scenarios/Scenario Outlines from a Test Execution).

Image Removed

One of the tests fails (on purpose).

The execution screen details of the Test Run will provide overall status information and Gherkin statement-level results, therefore we can use it to analyze the failing test.

Image Removed  

A given example can be expanded to see all Gherkin statements and, if available, it is possible to see also the attached screenshot(s).

Image Removed

Image Removed

Note: in this case, the bug was on the Scenario Outline example which was using a valid username/password combination.

Results are reflected on the covered item (e.g. Story). On its issue screen, coverage now shows that the item is OK based on the latest testing results, that can also be tracked within the Test Coverage panel bellow. 

Image Removed  

Using Git or other VCS as master

You can edit your .feature files using your IDE outside of Jira (eventually storing them in your VCS using Git, for example) alongside with remaining test code.

...

A new Test Execution will be created (unless you originally exported the Scenarios/Scenario Outlines from a Test Execution).

Image Added

One of the tests fails (on purpose).

The execution screen details of the Test Run will provide overall status information and Gherkin statement-level results, therefore we can use it to analyze the failing test.

Image Added  

A given example can be expanded to see all Gherkin statements and, if available, it is possible to see also the attached screenshot(s).

Image Added

Image Added

Note: in this case, the bug was on the Scenario Outline example which was using a valid username/password combination.

Results are reflected on the covered item (e.g. Story). On the issue screen, coverage now shows that the item is OK based on the latest testing results which can also be tracked within the Test Coverage panel bellow. 

Image Added  

Using Git or other VCS as master

You can edit your .feature files using your IDE outside of Jira (eventually storing them in your VCS using Git, for example) alongside the remaining test code.

In any case, you'll need to synchronize your .feature files to Jira so that you can have visibility of them and report results against them.

The overall flow would be something like this:

1. look at the existing "requirement"/Story issue keys to guide your testing; keep their issue keys
2. specify Cucumber/Gherkin .feature files in your IDE supporting Cypress and store it in Git, for example
3. implement the code related to Gherkin statements/steps and store it in Git, for example
4. import/synchronize the .feature files to Xray to provision or update corresponding Test entities
5. export/generate .feature files from Jira, so that they contain references to Tests and requirements in Jira
6. checkout the Cypress related code from Git
7. run the tests in the CI
8. import the results back to Jira

Usually, you would start by having a Story, or similar (e.g. "requirement"), to describe the behavior of a certain feature and use that to drive your testing. 

Having those to guide testing, we could then move to Cypress to describe and implement the Cucumber test scenarios.

In Cypress, tests test related code is stored inside the cypress/integration directory, which itself contains several other directories. In this case, we've organized them as follows:

...

AfterwardAfterwards, you can export those features out of Jira, based on some criteria , so they are properly tagged with corresponding issue keys; this is important because results need to contain these references.

...

• use the UI
  • 
• use the REST API (more info here)

  • Code Block
    language bash
    #!/bin/bash rm -f features/*.feature curl -u admin:admin "http://jiraserver.example.com/rest/raven/1.0/export/test?keys=CALC-7905;CALC-7906&fz=true" -o features.zip unzip -o features.zip -d features

• use one of the available CI/CD plugins (e.g. see an example of Integration with Jenkins)

We For CI only purpose, we will export the features to a new temporary directory named features/ on the root folder of your Cypress project (we'll need to tell Cypress to use this folder). Please note that while implementing the tests, .feature files should be edited inside the cypress/integration/login folder, in this case;  

After being exported, the created .feature(s) will contain references to the Test issue keys, eventually prefixed (e.g. "TEST_") depending on an Xray global setting, and the covered "requirement" issue key,  if that's the case. The naming of these files is detailed in Export Cucumber Features.

...

After running the tests, results can be imported to Xray via the REST API, or the Import Execution Results action within the Test Execution, or by using one of the available CI/CD plugins (e.g. see an example of Integration with Jenkins).

#!/bin/bash

curl -H "Content-Type: application/json" -X POST -u admin:admin --data @"report.json" http://jiraserver.example.com/rest/raven/1.0/import/execution/cucumber

...

...