Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


 

Cucumber is a framework for behavior-driven development (BDD), where the test specification is done in natural language, while the implementation of each specification's phrase (i.e., step) is done in code . The step's code may be done in (e.g., Java, Ruby, C#, or any some other language in fact).

Xray provides the tooling for making tool to create the specification inside JIRA (Jira, although the implementation code itself is out outside of Xray's scope).  

Standard Cucumber workflow

In this workflow, Tests are created /and managed always in JIRAJira.   

Gliffy Diagram
namecucumber_tests_workflow
 
pagePin3


  1. Specify Cucumber tests in using natural language, in JIRAJira.
  2. Export Cucumber features from JIRA, Jira to the CI environment, using the REST API.
  3. Implement tests in code and commit them to the source code versioning sytemsystem.
  4. Execute tests in the CI environment.
  5. Report results to Xray, using the REST API.

Note that Cucumber steps and related code may, in fact, be implemented before the first step, or may not even not be needed to be implemented for a given Test if all the sentences are already implemented as code.

 


Info
titleLearn more

See Automated Tests (importImport/exportExport) for more info information on how Cucumber feature files are generated and how to import back results to Xray.


Legacy Cucumber workflow

This worfklow can apply The workflow below applies to legacy/existing projects , that already have Cucumber tests in some version control system (e.g., GIT, SVN), possibly alongside the software's implementation.

 There are also cases where users prefer to edit Cucumber tests in some other using another tool; even so, they can take advantage of Xray.

Existing Cucumber tests are initialy initially imported to JIRAJira, or they can even be imported at a regular basis, if needed. Editing of Cucumber the tests can then proceed only on JIRA side, or can be done simultaneously in JIRA the Jira side. While test editing can also occur simultaneously in Jira and in some other tool. However, it is recommended to keep the edition of tests it only on one side, preferably JIRAJira


Gliffy Diagram
namecucumber_tests_legacy_workflow
pagePin1


  1. Create Cucumber tests (e.g., Scenario/Scenario Outlines) and their respective step code in some other tool, and the respective step's code.
  2. Import existing Cucumber tests using the REST API (e.g., from an existing code base which may have tests, and their corresponding step 's implementation) 
  3. Optionally, and in parallel, 
    1. Specify additional or edit existing Cucumber tests in natural language, in JIRAJira.
    2. Implement remaining tests in code and commit them to the source code versioning sytemsystem.
  4. Export Cucumber features from JIRA, Jira to the CI environment, using the REST API or the Jenkins/Bamboo plugins.
  5. Execute tests in the CI environment.
  6. Report results to Xray, using the REST API or the Jenkins/Bamboo plugins.
  7. Repeat from step 3 onwards.


Note
titlePlease note

As of Xray v2.3.x, step 2 is not idempotent, i.e., if you try to import Cucumber Tests multiple times using the endpoint Importing Cucumber Tests - REST, you may end up with repeated entities in Jira.


Common problems

Submission of Cucumber results says "No tests found in execution result"

Most probably you're trying to submit a Cucumber JSON results file without having previously created the Cucumber Tests (e.g., Scenario/Scenario Outline) in Jira.

You need to first have these Test issues created in Jira, with the correct specification. Then, you need to export those Tests to one or more Cucumber .feature files, so you're able to run them in your CI environment. 

Following this workflow is important because whenever you export the features from Jira, along with the corresponding Cucumber Scenarios/Scenario Outlines, the Scenarios will be tagged with labels that will allow Xray to map the results back to the proper entities in Jira.

Is it possible to create of Cucumber tests automatically when submitting Cucumber results?

No. It's not possible to properly recreate the Cucumber Scenario/Scenario Outline/Background from a Cucumber result file in JSON format.

Where is the master of information for Cucumber Tests?

Our recommendation is to manage it in Jira and always edit the Cucumber Scenarios/Scenario Outlines/Backgrounds there.