Versions Compared

Old Version 4

changes.mady.by.user Sergio Freire

Saved on

compared with

New Version 5

changes.mady.by.user Sergio Freire

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
languagejava
titlesrc/test/java/calculator/StepDefinitions.java
collapsetrue
package calculator;

import io.cucumber.java.en.Given;
import io.cucumber.java.en.Then;
import io.cucumber.java.en.When;
import com.xray.tutorials.Calculator;

import static org.junit.Assert.*;

public class StepDefinitions {
    private Integer int1;
    private Integer int2;
    private Integer result;


    @Given("I have entered {int} into the calculator")
    public void i_have_entered_into_the_calculator(Integer int1) {
        this.int2 = this.int1;
        this.int1 = int1;
    }
    
    
    @When("I press add")
    public void i_press_add() {
        this.result =  Calculator.Add(this.int1, this.int2);
    }

    @When("I press multiply")
    public void i_press_multiply() {
        this.result =  Calculator.Multiply(this.int1, this.int2);
    }


    @Then("the result should be {int} on the screen")
    public void the_result_should_be_on_the_screen(Integer int1) {
        // Write code here that turns the phrase above into concrete actions
        assertEquals((Integer)(this.int1 + this.int2), this.result);
    }
    

}
  • the test runner is defined in the RunCucumberTest class. Cucumber options can be overriden from the command line, whenever executing Maven.
Code Block
languagejava
titlesrc/test/java/calculator/RunCucumberTest.java
collapsetrue
package calculator;

import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(plugin = {"pretty"})
public class RunCucumberTest {

}


You can then export the specification of the test to a Cucumber .feature file via the REST API, or the Export to Cucumber UI action from within the Test/Test Execution issue or even based on an existing saved filter. As source, you can identify Test, Test Set, Test Execution, Test Plan or "requirement" issues. A plugin for your CI tool of choice can be used to ease this task.

So, you can either:

  • use one of the available CI/CD plugins (e.g. see details of Integration with Jenkins)
    • Image Added
  • use the REST API directly (more info here)
    • Code Block
      languagebash
      #!/bin/bash
 
rm -f features/*.feature
curl -u admin:admin  "http://jiraserver.example.com/rest/raven/1.0/export/test?keys=CALC-7931;CALC-7935&fz=true" -o features.zip
unzip -o features.zip  -d features
  • ... or even use the UI (e.g. from a Test issue)
    • Image Added


We will export the features to a new directory named features/ on the root folder of your Java project (we'll need to tell Maven to use this folder).


After being exported, the created .feature(s) will contain references to the Test issue key, eventually prefixed (e.g. "TEST_") depending on an Xray global setting, and the covered "requirement" issue key,  if that's the case. The naming of these files is detailed in Export Cucumber Features.

Code Block
titlefeatures/1_CALC-7931.feature
collapsetrue
@REQ_CALC-7931
Feature: As a user, I can calculate the sum of two numbers
	#As a user, I can calculate the sum of two numbers

	#Tests As a user, I can calculate the sum of two numbers
	@TEST_CALC-7934
	Scenario Outline: sum of two positive numbers
		Given I have entered <input_1> into the calculator
		And I have entered <input_2> into the calculator
		When I press <button>
		Then the result should be <output> on the screen
		
		  Examples:
		    | input_1 | input_2 | button | output |
		    | 20      | 30      | add    | 50     |
		    | 2       | 5       | add    | 7      |
		    | 0       | 40      | add    | 40     |
		    | 4       | 50      | add    | 54     |
		    | 5       | 50      | add    | 55     |	

	
	@TEST_CALC-7933
	Scenario: negative integer adition
		Given I have entered -1 into the calculator
		And I have entered 2 into the calculator
		When I press add
		Then the result should be 1 on the screen	

	#Tests As a user, I can calculate the sum of two numbers
	@TEST_CALC-7932
	Scenario: simple integer addition
		Given I have entered 1 into the calculator
		And I have entered 2 into the calculator
		When I press add
		Then the result should be 3 on the screen
Code Block
titlefeatures/2_CALC-7935.feature
collapsetrue
@REQ_CALC-7935
Feature: As a user, I can multiply two numbers
	#As a user, I can multiply two numbers

	#simple integer multiplication
	@TEST_CALC-7936
	Scenario: simple integer multiplication
		Given I have entered 3 into the calculator
		And I have entered 0 into the calculator
		When I press multiply
		Then the result should be 0 on the screen



To run the tests and produce a Cucumber JSON report, we can run Maven and specify that we want a report in Cucumber JSON format and that it should process .features from the features/ directory.


Code Block
languagebash
mvn compile test -Dcucumber.plugin="json:report.json" -Dcucumber.features="features/"


This will produce one Cucumber JSON report with all results.

After running the tests, results can be imported to Xray via the REST API, or the Import Execution Results action within an existing Test Execution, or by using one of the available CI/CD plugins (e.g. see an example of Integration with Jenkins).

Code Block
languagebash
titleexample of a Bash script to import results using the standard Cucumber endpoint
collapsetrue
curl -H "Content-Type: application/json" -X POST -u admin:admin --data @"report.json" http://jiraserver.example.com/rest/raven/1.0/import/execution/cucumber

Image Added


Info
titleWhich Cucumber endpoint to use?

To import results, you can use two different endpoints/"formats" (endpoints described in Import Execution Results - REST):

  1. the "standard cucumber" endpoint
  2. the "multipart cucumber" endpoint

The standard cucumber endpoint (i.e. /import/execution/cucumber) is simpler but more restrictive: you cannot specify values for custom fields on the Test Execution that will be created.  This endpoint creates new Test Execution issues unless the Feature contains a tag having an issue key of an existing Test Execution.

The multipart cucumber endpoint will allow you to customize fields (e.g. Fix Version, Test Plan), if you wish to do so, on the Test Execution that will be created. Note that this endpoint always creates new Test Executions (as of Xray v4.2).


In sum, if you want to customize the Fix Version, Test Plan and/or Test Environment of the Test Execution issue that will be created, you'll have to use the "multipart cucumber" endpoint.


A new Test Execution will be created (unless you originally exported the Scenarios/Scenario Outlines from a Test Execution).

Image Added


One of the tests fails (on purpose).

The execution screen details of the Test Run will provide overall status information and Gherkin statement-level results, therefore we can use it to analyze the failing test.

Image Added

Results, including for each example on Scenario Outline, can be expanded to see all Gherkin statements.

Image Added  Image Added


Note: in this case, the bug was added on purpose on the Calculator class. 


Code Block
languagejava
titlebuggy Multiply() method in Calculator.java
collapsetrue
public static int Multiply(int num1, int num2 )
{
    if ((num1==1) || (num2==1)) {
        return 0;
    } else {
        return num1 * num2;
    }
}
Info
titleScreenshots and other attachments

If available, it is possible to see also attached screenshot(s). For this, you'll need to use Cucumber's API and do it in a After hook, for example (using scenario.embed()).

The iconImage Added represents the evidences ("embeddings") for each Hook, Background and Steps.

Image Added


Results are reflected on the covered items (e.g. Story issues) and can be seen in ther issue screen.

Coverage now shows that the addition related user story (e.g. CALC-7931) is OK based on the latest testing results; on the other hand, the multiplication related user story (CALC-7935) is NOK since it has one test currently failing.

Image Added  Image Added


If we fix the code on the Calculator class, run the tests and import results, coverage for the multiplication related user story will be shown as OK.

Code Block
languagejava
titlefix of Multiply() method in Calculator.java
collapsetrue
public static int Multiply(int num1, int num2 )
{
	return num1 * num2;
}

Image Added


Using Git or other VCS as master

You can edit your .feature files using your IDE outside of Jira (eventually storing them in your VCS using Git, for example) alongside with remaining test code.

In any case, you'll need to synchronize your .feature files to Jira so that you can have visibility of them and report results against them.


The overall flow would be something like this:

  1. look at the existing "requirement"/Story issue keys to guide your testing; keep their issue keys
  2. specify Cucumber/Gherkin .feature files in your IDE supporting Cucumber/Gherkin and store it in Git, for example. Meanwhile, you may decide to import/synchronize them Xray to provision or update corresponding Test and/or Precondition entities
  3. implement the code related to Gherkin statements/steps and store it in Git, for example. 
  4. commit code and .feature file(s) to Git
  5. checkout the code from Git
  6. import/synchronize the .feature files to Xray to provision or update corresponding Test and/or Precondition entities
  7. export/generate .feature files from Jira, so that they contain references to Tests and requirements in Jira
  8. run the tests in the CI
  9. obtain the report in Cucumber JSON format
  10. import the results back to Jira


Image Added


Note that steps (5-10) performed by the CI tool are all automated, obviously.

To import .features to Jira we can either use the REST API or a CI tool. To export tagged .features from Jira, we can do it directly from Jira, by the REST API or using a CI tool; we'll see that ahead in more detail.


Example

All starts with a user story or some sort of “requirement” that you wish to validate. This is materialized as a Jira issue and identified by the corresponding issue key (e.g. CALC-7931).

Image Added

We can promptly check that it is “UNCOVERED” (i.e. that it has no tests covering it, no matter their type/approach).


Having those to guide testing, we could then describe and implement the Cucumber test scenarios using our favourite IDE.

Image Added


The related statement's code is managed outside of Jira and stored in Git, for example.

The tests related code is stored under src/test directory, which itself contains several other directories. In this case, they're organized as follows:

  • java/calculator: step implementation files and test runner class.
    • The steps "glue-code" is defined in the StepDefinitions class.
Code Block
languagejava
titlesrc/test/java/calculator/StepDefinitions.java
collapsetrue
package calculator;

import io.cucumber.java.en.Given;
import io.cucumber.java.en.Then;
import io.cucumber.java.en.When;
import com.xray.tutorials.Calculator;

import static org.junit.Assert.*;

public class StepDefinitions {
    private Integer int1;
    private Integer int2;
    private Integer result;


    @Given("I have entered {int} into the calculator")
    public void i_have_entered_into_the_calculator(Integer int1) {
        this.int2 = this.int1;
        this.int1 = int1;
    }
    
    
    @When("I press add")
    public void i_press_add() {
        this.result =  Calculator.Add(this.int1, this.int2);
    }

    @When("I press multiply")
    public void i_press_multiply() {
        this.result =  Calculator.Multiply(this.int1, this.int2);
    }


    @Then("the result should be {int} on the screen")
    public void the_result_should_be_on_the_screen(Integer int1) {
        // Write code here that turns the phrase above into concrete actions
        assertEquals((Integer)(this.int1 + this.int2), this.result);
    }
    

}
  • the test runner is defined in the RunCucumberTest class. Cucumber options can be overriden from the command line, whenever executing Maven.
Code Block
languagejava
titlesrc/test/java/calculator/RunCucumberTest.java
collapsetrue
package calculator;

import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(plugin = {"pretty"})
public class RunCucumberTest {

}


Before running the tests in the CI environment, you need to import your .feature files to Xray/Jira; you can invoke the REST API directly or use one of the available plugins/tutorials for CI tools.

Code Block
languagebash
titleexample of a shell script to import/synchronize .features to Jira and Xray
collapsetrue
zip -r features.zip src/test/resources/calculator/ -i \*.feature
curl -H "Content-Type: multipart/form-data" -u admin:admin -F "file=@features.zip" "http://jiraserver.example.com/rest/raven/1.0/import/feature?projectKey=CALC"

Image Added



Info
titlePlease note

Each Scenario of each .feature will be created as a Test issue that contains unique identifiers, so that if you import once again then Xray can update the existent Test and don't create any duplicated tests.


You can then export the specification of the test to a Cucumber .feature file via the REST API, or the Export to Cucumber UI action from within the Test/Test Execution issue or even based on an existing saved filter. As source, you can identify Test, Test Set, Test Execution, Test Plan or "requirement" issues. A plugin for your CI tool of choice can be used to ease this task.

...