In this tutorial, we will "create" some acceptance tests using Gauge framework and import back the results to Xray.
We'll reuse a sample Java maven-based project having tests that make use of Selenium to drive a browser in a sample shop application.
In Gauge, specifications are used to describe acceptance tests for business cases.
A specification is a business test case which describes a particular feature of the application that needs testing. (Gauge team)
Specifications may be done using steps or concepts (steps may be grouped together as a concept to define a unit of "business intent").
The idea is to have readable test scenarios and promote the reusability of steps/concepts.
Specifications typically live under a
A clear Gauge overview may be found in Gauge's documentation site.
- Inside the project folder, install Gauge's java plugin
- Inside the project folder, install the
- A sample web admin shop must be installed; you can follow the instructions provided in Gauge's sample project to download the .war file. After being downloaded, it may be easily started (it takes a few seconds to do so)
For this tutorial, we'll use a sample Java maven-based project provided by the Gauge team.
The project contains several tests, as part of distinct specifications.
Next, you may see an example of the
PlaceOrder.spec file, which contains two scenarios (i.e. tests): "Buy a book" and "Cart retains items until order is placed".
The steps implementation is done in different classes (you may freely structure these classes).
The following code shows the associated implementation for the previous specification.
Two things should be mentioned though:
- some steps (e.g. for login/logout) are implemented in other classes
- as you can see, the previous specification does not use these "raw" steps as such; instead, it uses concepts detailed in a specific file (PlaceOrder.cpt) as shown ahead
Since we're using a maven-based project, tests can be run as usual (i.e. "mvn test"); otherwise, you could use the "gauge run ..." command.
Note that we won't use Surefire to generate the JUnit XML report but we'll use Gauge's "xml-report" plugin instead.
After running the tests and generating the JUnit XML report (e.g., result.xml), it can be imported to Xray (either by the REST API or through Import Execution Results action within the Test Execution).
A Test Execution will be created containing information about the executed scenarios.
Each scenario is mapped to a Generic Test in Jira, and the Generic Test Definition field contains the name of the specification concatenated with the scenario name.
The Execution Details of the Generic Test contains information about the "Test Suite" (as per JUnit format), which in this case corresponds to the scenario name with a prefix.
Sometimes you may use data tables in your specifications.
In that case, the scenario will be executed multiple times (once per row) leading to multiple results in the JUnit XML report produced by the "xml-report" plugin.
This, in turn, will lead to multiple Test issues being created.
The unique identifier for the Test will be weak in the sense that it is composed of the specification plus the scenario name followed by a sequential number (i.e. row number).
In other words, if at some point in time you change the table rows order, results can be reported against the wrong existing Test issue. This may be avoided if you manually update the Generic Test Definition field with the proper row index before submitting results. A better way of identifying these scenarios could be perhaps achieved and agreed with the xml-report plugin team.