Overview

In this tutorial, we will create some tests in Cucumber using Java.

Cucumber is mainly a collaboration framework used in BDD context in order to improve shared understanding within the team, usually during "3 Amigos" sessions. That's its main fit.

However, some teams use it in other contexts (e.g. after sofware has being built) for implementing automated tests and take advantage of Gherkin syntax to have visibility/abstraction of the underlying automation code and have reusable automation code.

(Test) Scenarios derived from Cucumber are executable specifications; their statements will have a corresponding code implementation. These test scenarios are feature and more business oriented; they're not unit/integration tests.

Your specification is made using Gherkin (i.e. Given, When, That) statements in Scenario(s) or Scenario Outline(s), eventually complemented with a Background. Implementation of each Gherkin statement (i.e. "step") is done in code; the Cucumber framework finds the code based on regular or cucumber expressions.

Usage scenarios

Cucumber is used in diverse scenarios. Next you may find some usage patterns, even though Cucumber usage is mostly recommended only if you are adopting BDD.

1. Teams adopting BDD, start by defining a user story and clarify it using Cucumber Scenario(s); usualy, Cucumber Scenario(s)/Scenario Outline(s) are specified directly in Jira, using Xray
2. Teams adopting BDD but that favour a more Git based approach (e.g. GitOps). In this case, stories would be defined in Jira but Cucumber .feature files would be specified using some IDE and would be stored in Git, for example
3. Teams not adopting BDD but still using Cucumber, more as an automation framework. Sometimes focused on regression testing; sometimes, for non-regression testing. In this case, cucumber would be used...
   1. With a user story or some sort of "requirement" described in Jira
   2. Without any story/"requirement" described in Jira

You may be adopting, or aiming to, one of the previous patterns.

Before moving into the actual implementation, we need to decide which workflow we'll use: do we want to use Xray/Jira as the master for writing the declarative specification (i.e. the Gherkin based Scenarios), or do we want to manage those outside using some editor and store them in Git, for example?

Example

For the purpose of this tutorial, we'll use a simple, dummy Calculator implemented in a Java class as our target for testing.

package com.xray.tutorials;

public class Calculator
{
// Square function
public static int Square(int num)
{
    return num*num;
}
// Add two integers and returns the sum
public static int Add(int num1, int num2 )
{
    return num1 + num2;
}
// Add two integers and returns the sum
public static double Add(double num1, double num2 )
{
    return num1 + num2;
}
// Multiply two integers and retuns the result... this code is buggy on purpose
public static int Multiply(int num1, int num2 )
{
    if (num1==0) {
        return num2;
    } else if (num2==0) {
        return num1;
    } else {
        return num1 * num2;
    }
}

public static int Divide(int num1, int num2 )
{
    return num1 / num2;
}

// Subtracts small number from big number
public static int Subtract(int num1, int num2 )
{
    if ( num1 > num2 )
    {
    return num1 - num2;
    }
    return num2 - num1;
    }
}

This tutorial, has the following requirements:

• Java
• Add the dependency of cucumber-jvm (i.e. cucumber-java) to your maven "pom.xml" file

In case you need to interact with Xray REST API at low-level using scripts (e.g. Bash/shell scripts), this tutorial uses an auxiliary file with the credentials (more info in Global Settings: API Keys).

{ "client_id": "215FFD69FE4644728C72180000000000","client_secret": "1c00f8f22f56a8684d7c18cd6147ce2787d95e4da9f3bfb0af8f020000000000" }

Using Jira and Xray as master

This section assumes using Xray as master, i.e. the place that you'll be using to edit the specifications (e.g. the scenarios that are part of .feature files).

The overall flow would be something like this, assuming Git as the source code versioning system:

1. define the story (skip if you already have it)
2. create Scenario/Scenario Outline as a Test in Jira; usually, it would be linked to an existing "requirement"/Story (i.e. created from the respective issue screen)
3. implement the code related to Gherkin statements/steps and store it in Git, for example. To start, and during development, you may need to generate/export the .feature file to your local environment
4. commit previous code to Git
5. checkout the code from Git
6. generate .feature files based on the specification made in Jira
7. run the tests in the CI
8. obtain the report in Cucumber JSON format
9. import the results back to Jira

Note that steps (5-9) performed by the CI tool are all automated, obviously.

To generate .feature file(s) based on Scenarios defined in Jira (i.e. Cucumber Tests and Preconditions), we can do it directly from Jira, by the REST API or using a CI tool; we'll see that ahead in more detail.

Step-by-step

All starts with a user story or some sort of “requirement” that you wish to validate. This is materialized as a Jira issue and identified by the corresponding issue key (e.g. CALC-640).

We can promptly check that it is “UNCOVERED” (i.e. that it has no tests covering it, no matter their type/approach).

If you have this "requirement" as a Jira issue, then you can just use the "Create Test" on that issue to create the Scenario/Scenario Outline and have it automatically linked back to the Story/"requirement".

Otherwise, you can create the Test using the standard (issue) Create action from Jira's top menu. 

We need to create the Test issue first and fill out the Gherkin statements later on in the Test issue screen.

After the Test is created, and since we have done it from the user story screen, it will impact the coverage of related "requirement"/story.

The coverage and the test results can be tracked in the "requirement" side (e.g. user story). In this case, you may see that coverage changed from being UNCOVERED to NOTRUN (i.e. covered and with at least one test not run).

Additional tests could be created, eventually linked to the same Story or linked to another one (e.g. multiplication).

The related statement's code is managed outside of Jira and stored in Git, for example.

The tests related code is stored under src/test directory, which itself contains several other directories. In this case, they're organized as follows:

• java/calculator: step implementation files and test runner class.
  • The steps "glue-code" is defined in the StepDefinitions class.

package calculator;

import io.cucumber.java.en.Given;
import io.cucumber.java.en.Then;
import io.cucumber.java.en.When;
import com.xray.tutorials.Calculator;

import static org.junit.Assert.*;

public class StepDefinitions {
    private Integer int1;
    private Integer int2;
    private Integer result;


    @Given("I have entered {int} into the calculator")
    public void i_have_entered_into_the_calculator(Integer int1) {
        this.int2 = this.int1;
        this.int1 = int1;
    }
    
    
    @When("I press add")
    public void i_press_add() {
        this.result =  Calculator.Add(this.int1, this.int2);
    }

    @When("I press multiply")
    public void i_press_multiply() {
        this.result =  Calculator.Multiply(this.int1, this.int2);
    }

    @Then("the result should be {int} on the screen")
    public void the_result_should_be_on_the_screen(Integer value) {
        assertEquals(value, this.result);
    }
    
}

• the test runner is defined in the RunCucumberTest class. Cucumber options can be overriden from the command line, whenever executing Maven.

package calculator;

import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(plugin = {"pretty"})
public class RunCucumberTest {

}

You can then export the specification of the test to a Cucumber .feature file via the REST API, or the Export to Cucumber UI action from within the Test/Test Execution issue or even based on an existing saved filter. As source, you can identify Test, Test Set, Test Execution, Test Plan or "requirement" issues. A plugin for your CI tool of choice can be used to ease this task.

So, you can either:

• use one of the available CI/CD plugins (e.g. see details of Integration with Jenkins)
  • 
• use the REST API directly (more info here)
  • example of a shell script to export/generate .features from Xray  Expand source

    #!/bin/bash
     
    token=$(curl -H "Content-Type: application/json" -X POST --data @"cloud_auth.json" https://xray.cloud.getxray.app/api/v2/authenticate| tr -d '"')
    curl -H "Content-Type: application/json" -X GET -H "Authorization: Bearer $token" "https://xray.cloud.getxray.app/api/v2/export/cucumber?keys=CALC-640;CALC-641" -o features.zip
     
    rm -rf features/*.feature
    unzip -o features.zip -d features

• ... or even use the UI (e.g. from a Test issue)
  • 

We will export the features to a new directory named features/ on the root folder of your Java project (we'll need to tell Maven to use this folder).

After being exported, the created .feature(s) will contain references to the Test issue key, eventually prefixed (e.g. "TEST_") depending on an Xray global setting, and the covered "requirement" issue key,  if that's the case. The naming of these files is detailed in Generate Cucumber Features.

@REQ_CALC-640
Feature: As a user, I can calculate the sum of 2 numbers
	#As a user, I can calculate the sum of 2 numbers

	#simple integer addition
	@TEST_CALC-642
	Scenario: simple integer addition
		Given I have entered 1 into the calculator
		And I have entered 2 into the calculator
		When I press add
		Then the result should be 3 on the screen

	#negative integer addition
	@TEST_CALC-643
	Scenario: negative integer addition
		Given I have entered -1 into the calculator
		And I have entered 2 into the calculator
		When I press add
		Then the result should be 1 on the screen

	#sum of two positive numbers
	@TEST_CALC-644
	Scenario Outline: sum of two positive numbers
		Given I have entered <input_1> into the calculator
		And I have entered <input_2> into the calculator
		When I press <button>
		Then the result should be <output> on the screen
		
		  Examples:
		    | input_1 | input_2 | button | output |
		    | 20      | 30      | add    | 50     |
		    | 2       | 5       | add    | 7      |
		    | 0       | 40      | add    | 40     |
		    | 4       | 50      | add    | 54     |
		    | 5       | 50      | add    | 55     |

@REQ_CALC-641
Feature: As a user, I can multiply two numbers
	#As a user, I can multiply two numbers

	#simple integer multiplication
	@TEST_CALC-645
	Scenario: simple integer multiplication
		Given I have entered 3 into the calculator
		And I have entered 0 into the calculator
		When I press multiply
		Then the result should be 0 on the screen

To run the tests and produce a Cucumber JSON report, we can run Maven and specify that we want a report in Cucumber JSON format and that it should process .features from the features/ directory.

mvn compile test -Dcucumber.plugin="json:report.json" -Dcucumber.features="features/"

mvn compile test -Dcucumber.plugin="json:report.ndjson" -Dcucumber.features="features/"
cat report.ndjson | cucumber-json-formatter --format ndjson > report.json

This will produce one Cucumber JSON report with all results.

After running the tests, results can be imported to Xray via the REST API, or the Import Execution Results action within an existing Test Execution, or by using one of the available CI/CD plugins (e.g. see an example of Integration with Jenkins).

#!/bin/bash
 
BASE_URL=https://xray.cloud.getxray.app
token=$(curl -H "Content-Type: application/json" -X POST --data @"cloud_auth.json" "$BASE_URL/api/v2/authenticate"| tr -d '"')
curl -H "Content-Type: application/json" -X POST -H "Authorization: Bearer $token"  --data @"merged-test-results.json" "$BASE_URL/api/v2/import/execution/cucumber"

A new Test Execution will be created (unless you originally exported the Scenarios/Scenario Outlines from a Test Execution).

One of the tests fails (on purpose).

The execution screen details of the Test Run will provide overall status information and Gherkin statement-level results, therefore we can use it to analyze the failing test.

Results, including for each example on Scenario Outline, can be expanded to see all Gherkin statements.

Note: in this case, the bug was added on purpose on the Calculator class. 

public static int Multiply(int num1, int num2 )
{
    if (num1==0) {
        return num2;
    } else if (num2==0) {
        return num1;
    } else {
        return num1 * num2;
    }
}

Results are reflected on the covered items (e.g. Story issues) and can be seen in ther issue screen.

Coverage now shows that the addition related user story (e.g. CALC-640) is OK based on the latest testing results; on the other hand, the multiplication related user story (CALC-641) is NOK since it has one test currently failing.

If we fix the code on the Calculator class, run the tests and import results, coverage for the multiplication related user story will be shown as OK.

public static int Multiply(int num1, int num2 )
{
	return num1 * num2;
}

Using Git or other VCS as master

You can edit your .feature files using your IDE outside of Jira (eventually storing them in your VCS using Git, for example) alongside with remaining test code.

In any case, you'll need to synchronize your .feature files to Jira so that you can have visibility of them and report results against them.

The overall flow would be something like this:

1. look at the existing "requirement"/Story issue keys to guide your testing; keep their issue keys
2. specify Cucumber/Gherkin .feature files in your IDE supporting Cucumber/Gherkin and store it in Git, for example. Meanwhile, you may decide to import/synchronize them Xray to provision or update corresponding Test and/or Precondition entities
3. implement the code related to Gherkin statements/steps and store it in Git, for example. 
4. commit code and .feature file(s) to Git
5. checkout the code from Git
6. import/synchronize the .feature files to Xray to provision or update corresponding Test and/or Precondition entities
7. export/generate .feature files from Jira, so that they contain references to Tests and requirements in Jira
8. run the tests in the CI
9. obtain the report in Cucumber JSON format
10. import the results back to Jira

Note that steps (5-10) performed by the CI tool are all automated, obviously.

To import .features to Jira we can either use the REST API or a CI tool. To export tagged .features from Jira, we can do it directly from Jira, by the REST API or using a CI tool; we'll see that ahead in more detail.

Step-by-step

All starts with a user story or some sort of “requirement” that you wish to validate. This is materialized as a Jira issue and identified by the corresponding issue key (e.g. CALC-640).

We can promptly check that it is “UNCOVERED” (i.e. that it has no tests covering it, no matter their type/approach).

Having those to guide testing, we could then describe and implement the Cucumber test scenarios using our favourite IDE.

The related statement's code is managed outside of Jira and stored in Git, for example.

The tests related code is stored under src/test directory, which itself contains several other directories. In this case, they're organized as follows:

• java/calculator: step implementation files and test runner class.
  • The steps "glue-code" is defined in the StepDefinitions class.

package calculator;

import io.cucumber.java.en.Given;
import io.cucumber.java.en.Then;
import io.cucumber.java.en.When;
import com.xray.tutorials.Calculator;

import static org.junit.Assert.*;

public class StepDefinitions {
    private Integer int1;
    private Integer int2;
    private Integer result;

    @Given("I have entered {int} into the calculator")
    public void i_have_entered_into_the_calculator(Integer int1) {
        this.int2 = this.int1;
        this.int1 = int1;
    }
    
    @When("I press add")
    public void i_press_add() {
        this.result =  Calculator.Add(this.int1, this.int2);
    }

    @When("I press multiply")
    public void i_press_multiply() {
        this.result =  Calculator.Multiply(this.int1, this.int2);
    }

    @Then("the result should be {int} on the screen")
    public void the_result_should_be_on_the_screen(Integer value) {
        assertEquals(value, this.result);
    }

}

• the test runner is defined in the RunCucumberTest class. Cucumber options can be overriden from the command line, whenever executing Maven.

package calculator;

import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(plugin = {"pretty"})
public class RunCucumberTest {

}

Before running the tests in the CI environment, you need to import your .feature files to Xray/Jira; you can invoke the REST API directly or use one of the available plugins/tutorials for CI tools.

#!/bin/bash
BASE_URL=https://xray.cloud.getxray.app
PROJECT=CALC

rm -f features.zip
zip -r features.zip src/test/resources/calculator/ -i \*.feature

token=$(curl -H "Content-Type: application/json" -X POST --data @"cloud_auth.json" "$BASE_URL/api/v2/authenticate"| tr -d '"')
curl -H "Content-Type: multipart/form-data" -H "Authorization: Bearer $token"  -F "file=@features.zip" "$BASE_URL/api/v2/import/feature?projectKey=$PROJECT"

You can then export the specification of the test to a Cucumber .feature file via the REST API, or the Export to Cucumber UI action from within the Test/Test Execution issue or even based on an existing saved filter. As source, you can identify Test, Test Set, Test Execution, Test Plan or "requirement" issues. A plugin for your CI tool of choice can be used to ease this task.

So, you can either:

• use one of the available CI/CD plugins (e.g. see details of Integration with Jenkins)
  • 
• use the REST API directly (more info here)
  • example of a shell script to export/generate .features from Xray  Expand source

    #!/bin/bash
     
    token=$(curl -H "Content-Type: application/json" -X POST --data @"cloud_auth.json" https://xray.cloud.getxray.app/api/v2/authenticate| tr -d '"')
    curl -H "Content-Type: application/json" -X GET -H "Authorization: Bearer $token" "https://xray.cloud.getxray.app/api/v2/export/cucumber?keys=CALC-640;CALC-641" -o features.zip
     
    rm -rf features/*.feature
    unzip -o features.zip -d features

• ... or even use the UI (e.g. from a Test issue)
  • 

We will export the features to a new directory named features/ on the root folder of your Java project (we'll need to tell Maven to use this folder).

After being exported, the created .feature(s) will contain references to the Test issue key, eventually prefixed (e.g. "TEST_") depending on an Xray global setting, and the covered "requirement" issue key,  if that's the case. The naming of these files is detailed in Generate Cucumber Features.

@REQ_CALC-640
Feature: As a user, I can calculate the sum of 2 numbers
	#As a user, I can calculate the sum of 2 numbers

	#simple integer addition
	@TEST_CALC-642
	Scenario: simple integer addition
		Given I have entered 1 into the calculator
		And I have entered 2 into the calculator
		When I press add
		Then the result should be 3 on the screen

	#negative integer addition
	@TEST_CALC-643
	Scenario: negative integer addition
		Given I have entered -1 into the calculator
		And I have entered 2 into the calculator
		When I press add
		Then the result should be 1 on the screen

	#sum of two positive numbers
	@TEST_CALC-644
	Scenario Outline: sum of two positive numbers
		Given I have entered <input_1> into the calculator
		And I have entered <input_2> into the calculator
		When I press <button>
		Then the result should be <output> on the screen
		
		  Examples:
		    | input_1 | input_2 | button | output |
		    | 20      | 30      | add    | 50     |
		    | 2       | 5       | add    | 7      |
		    | 0       | 40      | add    | 40     |
		    | 4       | 50      | add    | 54     |
		    | 5       | 50      | add    | 55     |

@REQ_CALC-641
Feature: As a user, I can multiply two numbers
	#As a user, I can multiply two numbers

	#simple integer multiplication
	@TEST_CALC-645
	Scenario: simple integer multiplication
		Given I have entered 3 into the calculator
		And I have entered 0 into the calculator
		When I press multiply
		Then the result should be 0 on the screen

To run the tests and produce a Cucumber JSON report, we can run Maven and specify that we want a report in Cucumber JSON format and that it should process .features from the features/ directory.

mvn compile test -Dcucumber.plugin="json:report.json" -Dcucumber.features="features/"

mvn compile test -Dcucumber.plugin="json:report.ndjson" -Dcucumber.features="features/"
cat report.ndjson | cucumber-json-formatter --format ndjson > report.json

This will produce one Cucumber JSON report with all results.

After running the tests, results can be imported to Xray via the REST API, or the Import Execution Results action within an existing Test Execution, or by using one of the available CI/CD plugins (e.g. see an example of Integration with Jenkins).

#!/bin/bash
 
BASE_URL=https://xray.cloud.getxray.app
token=$(curl -H "Content-Type: application/json" -X POST --data @"cloud_auth.json" "$BASE_URL/api/v2/authenticate"| tr -d '"')
curl -H "Content-Type: application/json" -X POST -H "Authorization: Bearer $token"  --data @"merged-test-results.json" "$BASE_URL/api/v2/import/execution/cucumber"

A new Test Execution will be created (unless you originally exported the Scenarios/Scenario Outlines from a Test Execution).

One of the tests fails (on purpose).

The execution screen details of the Test Run will provide overall status information and Gherkin statement-level results, therefore we can use it to analyze the failing test.

Results, including for each example on Scenario Outline, can be expanded to see all Gherkin statements.

Note: in this case, the bug was added on purpose on the Calculator class. 

public static int Multiply(int num1, int num2 )
{
    if ((num1==1) || (num2==1)) {
        return 0;
    } else {
        return num1 * num2;
    }
}

Results are reflected on the covered items (e.g. Story issues) and can be seen in ther issue screen.

Coverage now shows that the addition related user story (e.g. CALC-640) is OK based on the latest testing results; on the other hand, the multiplication related user story (CALC-641) is NOK since it has one test currently failing.

If we fix the code on the Calculator class, run the tests and import results, coverage for the multiplication related user story will be shown as OK.

public static int Multiply(int num1, int num2 )
{
	return num1 * num2;
}

FAQ and Recommendations

Please see this page.

References

• Code used in this tutorial, along with some auxiliary scripts
• Sample project cucumber-java-skeleton
• Official Cucumber documentation
• Cucumber installation instructions for Java
• Cucumber API
• Cucumber expressions
• Testing in BDD with Gherkin based frameworks (e.g. Cucumber)
• Automated Tests (Import/Export)
• Exporting Cucumber Tests - REST