Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Cucumber is a framework for behavior-driven development (BDD), where the test specification is done in the natural language, while the implementation of each specification's phase phrase (i.e., step) is done in code (e.g., Java, Ruby or some other language).

Xray provides the tool to create the specification inside Jira, although the implementation code itself is outside of Xray's scope.  

One key thing to note is that the current version of Xray only allows you to manage the Cucumber scenarios in Jira/Xray; thus, Jira will be used as the master of informationthat you must decide is: where do you want to manage the Cucumber scenarios? Do you want to use Jira as the master of information (recommended)? Or do you want to use Git, for example, as the master for your features and corresponding scenarios?


Info
titlePlease note

Having both Jira and the VCS (e.g. Git, SVN) as master of information for Cucumber Scenarios/Scenario Outlines can lead to synchronization issues because the scenarios edited in JIRA won't be commited automatically to the VCS to the original feature files used.

Thus, you can use a mix of the Standard and VCS workflows below but have in mind that the scenarios can be out of synch between what you have in Jira and what you have in the VCS.


Table of Contents

Standard Cucumber workflow

In this workflow, Tests are created and managed in Jira, making thus Jira will be the master for the Cucumber scenarios.

...

Info
titleLearn more

See Automated Tests (Import/Export) for more information on how Cucumber feature files are generated and how to import back results to Xray.

Pure VCS based workflow

In this workflow, the features are being "managed" in the source code VCS (versioning control system). This means that users will be editing features elsewhere, other than Jira, and they will want to synchronize the scenarios contained within those features to Jira. Therefore the master of information will be Git, SVN or whatever VCS is being used.


Gliffy Diagram
namecucumber_tests_legacy_vcs_workflow2
pagePin1


  1. Create Cucumber tests (e.g. Scenario/Scenario Outlines) and their respective step code in some other tool.
    1. add an unique tag to each Scenario/Scenario outline, in the form of "@id:xxx", in which xxx is a number (e.g. 1, 2, 3, ...)
  2. Import existing Cucumber tests using the REST API (e.g., from an existing code base which may have tests, and their corresponding step implementation) 
  3. Export Cucumber features from Jira to the CI environment, using the REST API or the Jenkins/Bamboo plugins. It is crucial to use these exported features instead of the ones from the VCS, since the features exported from JIRA will contain additional tags that will allow several things, including the ability to correctly report the results to back to the correct entities in Jira, afterwards.
  4. Execute tests in the CI environment.
  5. Report results to Xray, using the REST API or the Jenkins/Bamboo plugins.
  6. Repeat from step 1 onwards.


Info
titlePlease note

This flow will work fine as long as:

  • An unique tag "@id:xxx" is added to each Scenario/Scenario Outline within the file. The id just needs to be unique (e.g. sequential) within the feature file
  • The path of the feature file is not changed

Is any of the previous conditions is not satisfied then new and duplicated Tests, and Pre-Conditions, may be created.

See an example in Importing Cucumber Tests - REST.


Legacy Cucumber migraton workflow

The workflow below applies to legacy/existing projects that already have Cucumber tests in some version control system (e.g., GIT, SVN), possibly alongside the software's implementation.


Existing Cucumber tests are initially imported to Jira. Editing the tests can then proceed on the Jira side. While test editing can also occur simultaneously in Jira and in some other tool, it is highly recommended to keep it only on one side, preferably Jira.


Gliffy Diagram
namecucumber_tests_legacy_workflow
pagePin1


  1. Create Cucumber tests (e.g., Scenario/Scenario Outlines) and their respective step code in some other tool.
  2. Import existing Cucumber tests using the REST API (e.g., from an existing code base which may have tests, and their corresponding step implementation) 
  3. Optionally, and in parallel, 
    1. Specify additional or edit existing Cucumber tests in natural language, in Jira.
    2. Implement remaining tests in code and commit them to the source code versioning system.
  4. Export Cucumber features from Jira to the CI environment, using the REST API or the Jenkins/Bamboo plugins.
  5. Execute tests in the CI environment.
  6. Report results to Xray, using the REST API (or CI plugins, if available).
  7. Repeat from step 3 onwards.
Info
titleLearn more

See Automated Tests (Import/Export) for more information on how Cucumber feature files are generated and how to import back results to Xray.

Common problems

Submission of Cucumber results says "No tests found in execution result"

...

Our recommendation is to manage it in Jira and always edit the Cucumber Scenarios/Scenario Outlines/Backgrounds there; otherwise, it will be hard to set up. You can use your VCS (e.g. Git, SVN) as the master but don't try to use it and at the same time also use Jira as master.