In this tutorial, we will create some tests in Behave, which is a Cucumber variant for Python.
The test (specification) is initialy created in Jira as a Cucumber Test and afterwards, it is exported using the UI or the REST API.
We'll show how to use Behave JSON format and also how to generate a Cucumber JSON report, in case you need it.
- Install Behave
- Install PyHamcrest
After running the tests and generating the Behave JSON report (e.g., data.json), it can be imported to Xray via the REST API or the Import Execution Results action within the Test Execution.
The execution details displays the result of the Cucumber Scenario.
See the available endpoints for importing Behave's results in Import Execution Results - REST.
Testing in BDD with Gherkin based frameworks (e.g. Cucumber) details the typical workflow for Cucumber-related tests.
Using Cucumber JSON reports
Cucumber JSON reports are supported by many tools, including some results parsers used by some CI tools. Besides it, as of Xray v3.1, the internal support for Cucumber JSON is more complete giving, for example, the ability to see step level information.
Behave does not provide, as of 2018, the ability to generate compatible Cucumber JSON reports. However, it provides the mechanism to use custom formatters. Thus, we can make our own implementation of a Cucumber JSON formatter.
The following code is based on a sample code provided by an open-source contributor "fredizzimo" (see original code here), with a small changes to make it handle correctly the JSON serialization of status results. You may create this
cucumber_json.py at the root of your project.
In this example, we'll use a
demo.feature file inspired in two Behave tutorials. The feature file needs to have the proper tags to the Test issue keys and, optionally, to the Test Execution in case you want to enfor the results to be submited to that same Test Executon. You may generate this feature from the UI of the Test Execution issue screen, by using the REST API.
The corresponding steps implementation code lives in the following files.
After running the tests and generating the Cucumber JSON report (e.g., cucumber.json), it can be imported to Xray via the REST API or the Import Execution Results action within the Test Execution.
Import results via REST API
The execution page provides detailed information, which in this case includes the results for the different examples along with the respective step results.