In this tutorial, we will create some tests in Cucumber/Gherkin, using SpecFlow and C#.
There are some possible workflows related with Cucumber. In this tutorial, we assume that the the tests (specification) are initially created in Jira as a Cucumber Tests and exported afterwards using the UI or the REST API; that's what we call the "standard" workflow. If you prefer to manage the .feature and respective Scenarios outside of Jira, like in your own environment or in Git/SVN, would More info in Testing with Cucumber. |
CucumberJson.cshtml
report template provided in this pageAfter creating Cucumber Tests of type "Scenario" or "Scenario Outline", in Jira, you can export the specification of the test to a Cucumber .feature file via the REST API or the Export to Cucumber UI action from within the Test Execution issue.
The created file will be similar to the following:
@CALC-2250 @REQ_CALC-2247 Feature: Sum Operation #In order to avoid silly mistake # #As a math idiot # #I want to be told the sum of two numbers @TEST_CALC-2249 Scenario Outline: Add two positive numbers Given I have entered <input_1> into the calculator And I have also entered <input_2> into the calculator When I press <button> Then the result should be <output> on the screen Examples: | input_1 | input_2 | button | output | | 20 | 30 | add | 50 | | 2 | 5 | add | 7 | | 0 | 40 | add | 40 | | 4 | 50 | add | 54 | | 5 | 50 | add | 55 | @TEST_CALC-2248 Scenario: add two numbers Given I have entered 50 into the calculator And I have also entered 70 into the calculator When I press add Then the result should be 120 on the screen @TEST_CALC-2251 Scenario Outline: add two negative numbers Given I have entered <input_1> into the calculator And I have also entered <input_2> into the calculator When I press <button> Then the result should be <output> on the screen Examples: | input_1 | input_2 | button | output | | -1 | -2 | add | -3 | | 1 | -1 | add | 0 | |
using System; using TechTalk.SpecFlow; using Microsoft.VisualStudio.TestTools.UnitTesting; using UnitTestProject1; namespace UnitTestProject1 { [Binding] public class CalculatorSteps { private int result; private Calculator calculator = new Calculator(); [Given(@"I have entered (.*) into the calculator")] public void GivenIHaveEnteredIntoTheCalculator(int number) { calculator.FirstNumber = number; } [Given(@"I have also entered (.*) into the calculator")] public void GivenIHaveAlsoEnteredIntoTheCalculator(int number) { calculator.SecondNumber = number; } [When(@"I press add")] public void WhenIPressAdd() { result = calculator.Add(); } [Then(@"the result should be (.*) on the screen")] public void ThenTheResultShouldBeOnTheScreen(int expectedResult) { Assert.AreEqual(expectedResult, result); } } } |
You have to use a proper template file in order to generate a valid Cucumber JSON report and you have to configure the test profile to use it.
@inherits TechTalk.SpecRun.Framework.Reporting.CustomTemplateBase<TestRunResult> @using System @using System.Collections.Generic @using System.Linq @using System.Globalization @using Newtonsoft.Json @using Newtonsoft.Json.Converters @using TechTalk.SpecRun.Framework @using TechTalk.SpecRun.Framework.Results @using TechTalk.SpecRun.Framework.TestSuiteStructure @using TechTalk.SpecRun.Framework.Tracing @{ var serializationSettings = new JsonSerializerSettings { ReferenceLoopHandling = ReferenceLoopHandling.Ignore, Converters = new List<JsonConverter>() { new StringEnumConverter(false) } }; var features = GetTextFixtures() .Select(f => new { description = "", elements = (from scenario in f.SubNodes let lastExecutionResult = GetTestItemResult(scenario.GetTestSequence().First()).LastExecutionResult() select new { description = "", id = "", keyword = "Scenario", line = scenario.Source.SourceLine + 1, name = scenario.Title, tags = scenario.Tags.Select(t => new { name = t, line = 1 }), steps = from step in lastExecutionResult.Result.TraceEvents where IsRelevant(step) && (step.ResultType == TestNodeResultType.Succeeded || step.ResultType == TestNodeResultType.Failed || step.ResultType == TestNodeResultType.Pending) && (step.Type == TraceEventType.Test || step.Type == TraceEventType.TestAct || step.Type == TraceEventType.TestArrange || step.Type == TraceEventType.TestAssert) let keyword = step.StepBindingInformation == null ? "" : step.StepBindingInformation.StepInstanceInformation == null ? "" : step.StepBindingInformation.StepInstanceInformation.Keyword let matchLocation = step.StepBindingInformation == null ? "" : step.StepBindingInformation.MethodName let name = step.StepBindingInformation == null ? "" : step.StepBindingInformation.Text let cucumberStatus = step.ResultType == TestNodeResultType.Succeeded ? "Passed" : step.ResultType.ToString() select new { keyword = keyword, line = 0, match = new { location = matchLocation }, name = name, result = new { duration = step.Duration.TotalMilliseconds, error_message = step.StackTrace, status = cucumberStatus } }, type = "scenario" }).ToList(), id = "", keyword = "Feature", line = f.Source.SourceLine + 1, tags = f.Tags.Select(t => new { name = t, line = 1 }), name = f.Title, uri = f.Source.SourceFile }); } @Raw(JsonConvert.SerializeObject(features, Formatting.Indented, serializationSettings)) |
<?xml version="1.0" encoding="utf-8"?> <TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5"> <Settings projectName="UnitTestProject1" projectId="{5359f4fc-ee65-45b2-bb4e-5c0255b88806}" /> <Execution stopAfterFailures="3" testThreadCount="1" testSchedulingMode="Sequential" /> <!-- For collecting by a SpecRun server update and enable the following element. For using the collected statistics, set testSchedulingMode="Adaptive" attribute on the <Execution> element. <Server serverUrl="http://specrunserver:6365" publishResults="true" /> --> <TestAssemblyPaths> <TestAssemblyPath>UnitTestProject1.dll</TestAssemblyPath> </TestAssemblyPaths> <DeploymentTransformation> <Steps> <!-- sample config transform to change the connection string--> <!--<ConfigFileTransformation configFile="App.config"> <Transformation> <![CDATA[<?xml version="1.0" encoding="utf-8"?> <configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform"> <connectionStrings> <add name="MyDatabase" connectionString="Data Source=.;Initial Catalog=MyDatabaseForTesting;Integrated Security=True" xdt:Locator="Match(name)" xdt:Transform="SetAttributes(connectionString)" /> </connectionStrings> </configuration> ]]> </Transformation> </ConfigFileTransformation>--> </Steps> </DeploymentTransformation> <Report> <Template name="CucumberJson.cshtml" outputName="data.json"/> </Report> </TestProfile> |
The tests can be run from within the IDE (e.g. Visual Studio) or by the command line; in the later case, make sure to specify the profile name and all the paths properly.
Since there is code-behind file generation, it is required to have the NuGet "SpecFlow.Tools.MsBuild.Generation" package.
msbuild /t:Clean;Rebuild cd bin\debug ..\..\..\packages\SpecRun.Runner.1.7.2\tools\SpecRun.exe run Default.srprofile /outputFolder:..\..\..\TestResults cd ..\.. |
After running the tests and generating the Cucumber JSON report (e.g., data.json), it can be imported to Xray via the REST API or the Import Execution Results action within the Test Execution.
curl -H "Content-Type: application/json" -X POST -u user:pass --data @"data.json" http://jiraserver.example.com/rest/raven/1.0/import/execution/cucumber |
Since the original feature was extracted from a Test Execution, the results will be updated on it (this happens because the .feature file contains the Test Execution's issue key as a tag).
If the .feature was created by hand, or managed elsewhere outside of Jira, and it didn't contain the Test Execution's key, then a brand new Test Execution would be created. |
The execution screen details will not only provide information on the overall test run result, but also of each of the examples provided in the Scenario Outline and on the respective steps.
Please see for an overview on how to use Cucumber Tests with Xray. |