Page History
...
The test (specification) is initialy created in Jira as a Cucumber Test and afterwards, it is exported using the UI or the REST API.
...
We'll show how to use Behave JSON format and also how to generate a Cucumber JSON report, in case you need it.
Table of Contents |
---|
Requirements
- Install Behave
- Install PyHamcrest
Code
Code Block | ||
---|---|---|
| ||
Feature: Showing off behave (tutorial01) @ABC-119 Scenario: Run a simple test Given we have behave installed When we implement a test Then behave will test it for us! |
...
After running the tests and generating the Behave JSON report (e.g., data.json), it can be imported to Xray via the REST API or the the Import Execution Results action action within the Test Execution.
...
The execution details displays the result of the Cucumber Scenario.
Info | ||
---|---|---|
| ||
See the available endpoints for importing Behave's results in Import Execution Results - REST. Testing with Cucumber details the typical workflow for Cucumber-related tests. |
Using Cucumber JSON reports
Info | ||
---|---|---|
| ||
Cucumber JSON reports are supported by many tools, including some results parsers used by some CI tools. Besides it, as of Xray v3.1, the internal support for Cucumber JSON is more complete giving, for example, the ability to see step level information. |
Behave does not provide, as of 2018, the ability to generate compatible Cucumber JSON reports. However, it provides the mechanism to use custom formatters. Thus, we can make our own implementation of a Cucumber JSON formatter.
The following code is based on a sample code provided by an open-source contributor "fredizzimo" (see original code here), with a small changes to make it handle correctly the JSON serialization of status results. You may create this cucumber_json.py
at the root of your project.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
# -*- coding: utf-8 -*-
from __future__ import absolute_import
from behave.formatter.base import Formatter
from behave.model_core import Status
import base64
import six
import copy
try:
import json
except ImportError:
import simplejson as json
# -----------------------------------------------------------------------------
# CLASS: JSONFormatter
# -----------------------------------------------------------------------------
class CucumberJSONFormatter(Formatter):
name = 'json'
description = 'JSON dump of test run'
dumps_kwargs = {}
json_number_types = six.integer_types + (float,)
json_scalar_types = json_number_types + (six.text_type, bool, type(None))
def __init__(self, stream_opener, config):
super(CucumberJSONFormatter, self).__init__(stream_opener, config)
# -- ENSURE: Output stream is open.
self.stream = self.open()
self.feature_count = 0
self.current_feature = None
self.current_feature_data = None
self._step_index = 0
self.current_background = None
self.current_background_data = None
def reset(self):
self.current_feature = None
self.current_feature_data = None
self._step_index = 0
self.current_background = None
# -- FORMATTER API:
def uri(self, uri):
pass
def status(self, status_obj):
if (status_obj == Status.passed):
return "passed"
elif (status_obj == Status.failed):
return "failed"
else:
return "skipped"
def feature(self, feature):
self.reset()
self.current_feature = feature
self.current_feature_data = {
'id': self.generate_id(feature),
'uri': feature.location.filename,
'line': feature.location.line,
'description': '',
'keyword': feature.keyword,
'name': feature.name,
'tags': self.write_tags(feature.tags),
'status': self.status(feature.status),
}
element = self.current_feature_data
if feature.description:
element['description'] = self.format_description(feature.description)
def background(self, background):
element = {
'type': 'background',
'keyword': background.keyword,
'name': background.name,
'location': six.text_type(background.location),
'steps': []
}
self._step_index = 0
self.current_background = element
def scenario(self, scenario):
if self.current_background is not None:
self.add_feature_element(copy.deepcopy(self.current_background))
element = self.add_feature_element({
'type': 'scenario',
'id': self.generate_id(self.current_feature, scenario),
'line': scenario.location.line,
'description': '',
'keyword': scenario.keyword,
'name': scenario.name,
'tags': self.write_tags(scenario.tags),
'location': six.text_type(scenario.location),
'steps': [],
})
if scenario.description:
element['description'] = self.format_description(scenario.description)
self._step_index = 0
@classmethod
def make_table(cls, table):
table_data = {
'headings': table.headings,
'rows': [ list(row) for row in table.rows ]
}
return table_data
def step(self, step):
s = {
'keyword': step.keyword,
'step_type': step.step_type,
'name': step.name,
'line': step.location.line,
'result': {
'status': 'skipped',
'duration': 0
}
}
if step.text:
s['doc_string'] = {
'value': step.text,
'line': step.text.line
}
if step.table:
s['rows'] = [{'cells': [heading for heading in step.table.headings]}]
s['rows'] += [{'cells': [cell for cell in row.cells]} for row in step.table]
if self.current_feature.background is not None:
element = self.current_feature_data['elements'][-2]
if len(element['steps']) >= len(self.current_feature.background.steps):
element = self.current_feature_element
else:
element = self.current_feature_element
element['steps'].append(s)
def match(self, match):
if match.location:
# -- NOTE: match.location=None occurs for undefined steps.
match_data = {
'location': six.text_type(match.location) or "",
}
self.current_step['match'] = match_data
def result(self, result):
self.current_step['result'] = {
'status': self.status(result.status),
'duration': int(round(result.duration * 1000.0 * 1000.0 * 1000.0)),
}
if result.error_message and result.status == 'failed':
# -- OPTIONAL: Provided for failed steps.
error_message = result.error_message
result_element = self.current_step['result']
result_element['error_message'] = error_message
self._step_index += 1
def embedding(self, mime_type, data):
step = self.current_feature_element['steps'][-1]
step['embeddings'].append({
'mime_type': mime_type,
'data': base64.b64encode(data).replace('\n', ''),
})
def eof(self):
"""
End of feature
"""
if not self.current_feature_data:
return
# -- NORMAL CASE: Write collected data of current feature.
self.update_status_data()
if self.feature_count == 0:
# -- FIRST FEATURE:
self.write_json_header()
else:
# -- NEXT FEATURE:
self.write_json_feature_separator()
self.write_json_feature(self.current_feature_data)
self.current_feature_data = None
self.feature_count += 1
def close(self):
self.write_json_footer()
self.close_stream()
# -- JSON-DATA COLLECTION:
def add_feature_element(self, element):
assert self.current_feature_data is not None
if 'elements' not in self.current_feature_data:
self.current_feature_data['elements'] = []
self.current_feature_data['elements'].append(element)
return element
@property
def current_feature_element(self):
assert self.current_feature_data is not None
return self.current_feature_data['elements'][-1]
@property
def current_step(self):
step_index = self._step_index
if self.current_feature.background is not None:
element = self.current_feature_data['elements'][-2]
if step_index >= len(self.current_feature.background.steps):
step_index -= len(self.current_feature.background.steps)
element = self.current_feature_element
else:
element = self.current_feature_element
return element['steps'][step_index]
def update_status_data(self):
assert self.current_feature
assert self.current_feature_data
self.current_feature_data['status'] = self.status(self.current_feature.status)
def write_tags(self, tags):
return [{'name': tag, 'line': tag.line if hasattr(tag, 'line') else 1} for tag in tags]
def generate_id(self, feature, scenario=None):
def convert(name):
return name.lower().replace(' ', '-')
id = convert(feature.name)
if scenario is not None:
id += ';'
id += convert(scenario.name)
return id
def format_description(self, lines):
description = '\n'.join(lines)
description = '<pre>%s</pre>' % description
return description
# -- JSON-WRITER:
def write_json_header(self):
self.stream.write('[\n')
def write_json_footer(self):
self.stream.write('\n]\n')
def write_json_feature(self, feature_data):
self.stream.write(json.dumps(feature_data, **self.dumps_kwargs))
self.stream.flush()
def write_json_feature_separator(self):
self.stream.write(",\n\n")
# -----------------------------------------------------------------------------
# CLASS: PrettyJSONFormatter
# -----------------------------------------------------------------------------
class PrettyCucumberJSONFormatter(CucumberJSONFormatter):
"""
Provides readable/comparable textual JSON output.
"""
name = 'json.pretty'
description = 'JSON dump of test run (human readable)'
dumps_kwargs = { 'indent': 2, 'sort_keys': True } |
In this example, we'll use a demo.feature
file inspired in two Behave tutorials. The feature file needs to have the proper tags to the Test issue keys and, optionally, to the Test Execution in case you want to enfor the results to be submited to that same Test Executon. You may generate this feature from the UI of the Test Execution issue screen, by using the REST API.
Code Block | ||
---|---|---|
| ||
@CALC-1958
Feature: demo
@TEST_CALC-1957
Scenario Outline: Use Blender with <thing>
Given I put "<thing>" in a blender
When I switch the blender on
Then it should transform into "<other thing>"
Examples: Amphibians
| thing | other thing |
| Red Tree Frog | mush |
| apples | apple juice |
@TEST_CALC-1956
Scenario: Run a simple test
Given we have behave installed
When we implement a test
Then behave will test it for us! |
The corresponding steps implementation code lives in the following files.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
# file:features/steps/blender.py
# -----------------------------------------------------------------------------
# DOMAIN-MODEL:
# -----------------------------------------------------------------------------
class Blender(object):
TRANSFORMATION_MAP = {
"Red Tree Frog": "mush",
"apples": "apple juice",
"iPhone": "toxic waste",
"Galaxy Nexus": "toxic waste",
}
def __init__(self):
self.thing = None
self.result = None
@classmethod
def select_result_for(cls, thing):
return cls.TRANSFORMATION_MAP.get(thing, "DIRT")
def add(self, thing):
self.thing = thing
def switch_on(self):
self.result = self.select_result_for(self.thing) |
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
# file:features/steps/step_tutorial01.py
# ----------------------------------------------------------------------------
# STEPS:
# ----------------------------------------------------------------------------
from behave import given, when, then
@given('we have behave installed')
def step_impl(context):
pass
@when('we implement a test')
def step_impl(context):
assert True is not False
@then('behave will test it for us!')
def step_impl(context):
assert context.failed is False |
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
# file:features/steps/step_tutorial03.py
# ----------------------------------------------------------------------------
# STEPS:
# ----------------------------------------------------------------------------
from behave import given, when, then
from hamcrest import assert_that, equal_to
from blender import Blender
@given('I put "{thing}" in a blender')
def step_given_put_thing_into_blender(context, thing):
context.blender = Blender()
context.blender.add(thing)
@when('I switch the blender on')
def step_when_switch_blender_on(context):
context.blender.switch_on()
@then('it should transform into "{other_thing}"')
def step_then_should_transform_into(context, other_thing):
assert_that(context.blender.result, equal_to(other_thing)) |
After running the tests and generating the Cucumber JSON report (e.g., cucumber.json), it can be imported to Xray via the REST API or the Import Execution Results action within the Test Execution.
Running tests
No Format |
---|
behave --format=cucumber_json:PrettyCucumberJSONFormatter -o cucumber.json --format=json -o behave.json features/demo.feature |
Import results via REST API
No Format |
---|
curl -H "Content-Type: application/json" -X POST -u user:password --data @cucumber.json https://sandbox.xpand-addons.com/rest/raven/1.0/import/execution/cucumber |
The execution page provides detailed information, which in this case includes the results for the different examples along with the respective step results.