What you'll learn

Learn Ranorex core concepts
Learn how creating test automation test cases looks like, at high-level, using Ranorex Studio
Run the tests and push the test report to Xray
Validate that the test results are available in Jira
Learn how to assess the impacts on related user stories in Jira, using Xray

Overview

Ranorex can be seen as a keyword-driven framework for implementing GUI test automation across a broad set of technologies, including desktop, web, and mobile. It's a codeless test automation solutiion. Ranorex provides comprehensive support for test automation as detailed in its core features. Data-driven testing is also supported.

Ranorex provides Ranorex Studio, the main application to implement and organize automated test scripts. Ranorex Studio has multiple components, including a recorder (Ranorex Recorder) and a object/element identifier (Ranorex Spy).

In this article we'll highlight some of Ranorex core concepts and see how to have visilbity of test automation results in Jira, using Xray.

Integrating with Xray is straightforward, using JUnit XML reports. This article hw

Ranorex concepts and mapping to Xray

Ranorex Studio provides a complete GUI for implementing automated tests. Therefore, we find some concepts typical in IDEs (e.g. solution, project).

There are some specifics though, related to test automation.

The only concept with a direct mapping to Xray will be Ranorex' Test case which will be abstracted as a Xray Test issue (unstructured/generic).

Ranorex concept	Description	Xray concept
Solution	In Ranorex Studio, a solution is the top-level container that contains all other test files. Solutions are organized into one or more projects. Whenever creating a solution, we may identify the type of application we aim to test (e.g. desktop, web, mobile). A solution has always a "test suite" project.
Project	A tailored place to organize test files. A project can be of one of several types, including "test suite", which offer different capabilities.
Test suite	The test suite is where you build, organize, and run your tests in Ranorex Studio. A test suite consists primarily of test cases.	Doesn't exist as an entity. Will be visible and part of the definition of each Test issue.
Test case	A test, composed of Modules, which in turn are composed of Actions.	Test issue.
Module	A modularized sequence of actions, that have a certain goal. Can be seen as a grouped sequence of steps in a test case. Modules can be reused between Test cases.
Action	A step, inside a Module. Can be a mouse/keyboard interaction or a validation.
Validation	An assertion, a validation.
Repository	A Repository contains Repository Items (i.e. UI elements) organized in a tree-like structure. UI elements that contain other UI elements are represented as folders in the repository, with app folders acting as top-level elements and rooted folders as children.
Repository item	A representation of a user interface (UI) element used in a test. Each repository item has a name and is defined by its RanoreXPath (i.e. path).


Overall test status	yes
Actions	no
Videos	no	perhaps
Data-driven

Implementing tests

We will implement some automated tests for a dummy web site, providing an authentication mechanism that we aim to check, namelly the login and logout features.

Our tests, part of a test suite, include:

valid login
invalid login
valid logout
invalid login (data-driven scenario)

Solution and project setup

We start by creating a solution, web-based one as we aim to test a web site.

Under "Additional options" we can customize the language of the underlying code that Ranorex will use to support the test automation.

A project of type "test suite" will be created.

It contains one Test Suite, where we can create and manage our test cases. We may rename this Test Suite to have a more meaningful name (e.g. "Authentication").

Implementing automated tests

To add some Test cases to the Test Suite, we can use "Add" button.

We can rename the Test case (i.e. from TestCase to whatever describes it). We then add a Setup and a Teardown section and add the OpenBrowser and CloseBrowser modules, to each section respectively. Opening the OpenBrowser module, by double-clicking on it, will allow us to set the URL to be usused.

We can then use Ranorex Recorder to create a "module" (i.e. a set of sequential actions and/or validations on UI elements).

This will be used to implement interactions with our web site, without having to code.

We choose "Record" adn then we're redirected to the browser, where we can perform actions which will be recorded.

While recording the actions of our module, we can remove some that may be added by accident for example. We can also pause and stop recording.

In order to implement a test, we need to add at least one validation.

For that, we choose "Validate" and move the mouse hover the element we want to validate and click on it.

A validation dialog enable us to perform multiple asserts at the same time, on a given element (e.g. element exists, is visible, contains a given text). Each assert/validation will be created as a "Validate" action in our module.

A possible "Valid Login" test case could be composed of a setup section to open the browser (i.e. using the "OpenBrowser" module), a recorded module where we enter the credentials and validate the welcome page, followed by the close browser instruction (as part of the CloseBrowser module).

The "Login_with_valid_credentials" module could be composed of these actions, depending on the actions you make or wish to make.

We may decide to implement additional tests, reusing existing modules.

As an example, a test case (i.e. "Valid_Logout") that checks the valid logout procedure can use the "Login_with_valid_credentials" module as the first macro step, before executing the module and its actions (i.e. "Logout") that perform the logout and verify its result.

Running the tests

Using Ranorex Studio

Test cases can be "run" from within the Ranorex Studio UI .

Before running, Ranorex may decide to build the current project which will produce a executable file. Tests are in fact performed by this executable file and not Ranorex Studio self.

Whenever running, Ranorex Studio uses a run configuration to know which test cases to run; in the previous screenshot it is called "All_tests" and contains all test cases, since they're selected.

From the command-line

Integrating with Xray

In order to have visibility of our test automation results in Jira, we need to generate a JUnit XML that can then be submitted ot Xray.

Enabling the JUnit XML report can be done right from Ranorex Studio or from the command-line.

After running the tests, we

If we run the tests from within the Ranorex Studio UI, we can go to the properties of our test suite to customize the report file name, the reports directory, and enable the JUnit XML report.

If we run the tests from the command-line, by calling the executable built by Ranorex, we can enable the JUnit XML report, customize the report file base file name, among other options.

example of a shell script to run the tests

MyTest1.exe /junit /reportfile:results

In this case, the report will be stored in the current directory and wil be named results.rxlog.junit.xml.

To submit the report to Xray, we can use our favourite CI/CD tool or a simple shell script for that.

example of a shell script to import results to Xray

curl -H "Content-Type: multipart/form-data" -u jira_username:jira_password -F "file=@results.rxlog.junit.xml" jiraserver_base_url.example.com/rest/raven/1.0/import/execution/junit?projectKey=DEMO

After submiting the test automation report, a Test Execution will be created in Jira, containing the results for each Test case.

A Test issue will be auto-provisioned, unless it already exists, per each Test Case.

The Test Suite name along with the Test Case name will be used as unique identifier for the Generic Test that will be created.

In the Test Run details of the corresponding Test in Xray, we can see this information along with the overall result. The original Test Suite name appears also as a context.

Tips

Seeing the impacts of test automation results on user stories or requirements

After uploading the test automation results, users can link the Test issues to existing user stories or requirements. That will enable users to track coverage and thus assess if user stories are covered by automated test scripts and if based on that, the corresponding user story can be considered to OK or NOK.

Please remember that coverage is an heuristic but that can still be quite helpful to assess the readiness of user stories, individually or at the release level.

Assuming we have a user story (new or existing), we can then link it to the Tests that correspond to the Test Cases implemented using Ranorex.

We can do that right from the user story issue screen, using the "Link" action.

Then, we select the Tests that were auto-provisioned earlier on upon the first import of the test report.

Finally, we can see the latest test results right from the user story issue screen along with the calculated coverage information for the user story; the latter is avaiable (both on the Requirement Status custom field and also on the Test Coverage panel.

Any additional imports of results, will appear automatically reflected on the user story issue screen as the Tests are already linked to the user story.

Run iterations and data-driven tests

Ranorex Studio has support for run iterations and data-driven tests.

These are two different concepts; while run interations are just a way to run the same test case multiple times by executing the exact same modules and actions, data-driven tests will impact on the action being performed (e.g. for exercising the same test case but with different inputs).

It's possible to have visibility of the correspoding test results in Jira using Xray but some.

Please note

Due to the way Ranorex Studio reports these results on the JUnit XML report, different Test issues will be created for each run iteration or data row.

We should have this in mind as it hardens management of these Test issues (e.g. number of unrelated Tests, linkage to user stories).

Run iterations

On the properties of a test case, we can configure the number of run iterations (i.e. iteration count).

After running the test case and importing the results to Xray using the JUnit XML report, a Test Execution with 3 Tests is created in Xray.

Due to the way run iterations are reported in the JUnit XML report, each run iteration for our test case is abstracted as a different Test issue, with the run iteration being part of its defintion.

In other words, we'll have as many Test issues as the iteration count configured for the test case.

Data-driven tests

To make a data-driven test we need to use some variables in our actions, instead of using hardcoded values/strings.

Then we need to configure a data-source for the selected test case.

There are different types of data-sources; a simple built-in data table can be used to specify several named columns and some rows of data for them.

We then need to bind the columns of our data-source to the variables used in the previous module.

We can run the test case in Ranorex Studio and see the results for each data row.

After running the test case and importing the results to Xray using the JUnit XML report, a Test Execution having 3 Tests related to our data-source is created in Xray.