Cucumber is a framework for behavior-driven development (BDD), where the test specification is done in the natural language, while the implementation of each specification's phase (i.e., step) is done in code (e.g., Java, Ruby or some other language).
Xray provides the tool to create the specification inside Jira, although the implementation code itself is outside of Xray's scope.
One key thing to note is that the current version of Xray only allows you to manage the Cucumber scenarios in Jira/Xray; thus, Jira will be used as the master of information.
Standard Cucumber workflow
In this workflow, Tests are created and managed in Jira, making Jira the master for the Cucumber scenarios.
- Specify Cucumber tests using natural language, in Jira.
- Export Cucumber features from Jira to the CI environment, using the REST API.
- Implement tests in code and commit them to the source code versioning system.
- Execute tests in the CI environment.
- Report results to Xray, using the REST API.
Note that Cucumber steps and related code may, in fact, be implemented before the first step, or may not even be needed to be implemented for a given Test if all the sentences are already implemented as code.
Learn more
See Automated Tests (Import/Export) for more information on how Cucumber feature files are generated and how to import back results to Xray.
Common problems
Submission of Cucumber results says "No tests found in execution result"
You're probably trying to submit a Cucumber JSON results file without having previously created the Cucumber Tests (e.g., Scenario/Scenario Outline) in Jira.
You need to first have these Test issues created in Jira, with the correct specification. Then, you need to export those Tests to one or more Cucumber .feature files, so you're able to run them in your CI environment.
Following this workflow is important because whenever you export the features from Jira, along with the corresponding Cucumber Scenarios/Scenario Outlines, the Scenarios will be tagged with labels that will allow Xray to map the results back to the proper entities in Jira.
Is it possible to automatically create Cucumber tests when submitting Cucumber results?
No. It's not possible to properly recreate the Cucumber Scenario/Scenario Outline/Background from a Cucumber result file in JSON format.
Where is the master of information for Cucumber Tests?
Our recommendation is to manage it in Jira and always edit the Cucumber Scenarios/Scenario Outlines/Backgrounds there; otherwise, it will be hard to set up.