Versions Compared

Old Version 1

changes.mady.by.user Sergio Freire

Saved on

compared with

New Version 2

changes.mady.by.user Sergio Freire

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Overview

Robot Framework is a tool used by teams adopting ATDD(Acceptance Test Driven Development).

Broadly speaking, it can be used to automate acceptance “test cases” (i.e. scripts) no matter the moment you decide to do so or the practices your team follows even though it's preferable to do it at start, involving the whole team in order to pursue shared understanding.

In in this tutorial, we will execute specify some tests using the using Robot Framework assuming that your team is adopting ATDD.

This tutorial explores the specific integration Xray provides for Robot Framework XML reports.


Info

You may find the full source for this example in this GitHub repository.


Requirements

  • Robot Framework
  • SeleniumLibrary (previously Selenium2Library)
  • Java (if using the Java variant of the "robot frameworkRobot Framework")

Description

Below are several Robot test suites, each one containing several Robot test cases. 

Code Block
titlelogin_tests/gherkin_login.robot
collapsetrue
*** Settings ***
Documentation A test suite with a single Gherkin style test.
...
... This test is functionally identical to the example in
... valid_login.robot file.
Resource resource.robot
Test Teardown Close Browser
*** Test Cases ***
Gherkin Valid Login
 [Tags] CALC-1 CALC-42
 Given browser is opened to login page
 When user "admin" logs in with password "admin"
 Then welcome page should be open
*** Keywords ***
Browser is opened to login page
 Open browser to login page
User "${username}" logs in with password "${password}"
 Input username ${username}
 Input password ${password}
 Submit credentials

...

titlelogin_tests/invalid_login.robot
collapsetrue

...

If the team is adopting ATDD and working collaboratively in order to have a shared understanding of what is going to be developed, why and some concrete examples of usage, then the flow would be something similar to the following diagram.

Image Added



All starts with a user story or some sort of “requirement” that you wish to validate. This is materialized as a Jira issue and identified by the corresponding issue key (e.g. ROB-11).

Image Added

We can promptly check that it is “UNCOVERED” (i.e. that it has no tests covering it, no matter their type/approach).

A Test Plan can be created to define the scope of the testing that we aim to perform, group, and consolidate the corresponding results. Besides the user story, we may also add the Test Plan to the Board and assign it explicitly to a sprint. This will increase visibility of testing progress and help closing the gap between dev<>testers.


Image Added   Image Added


A tester/SDET could simply focus on implementing the automated test cases:

  • The tester would write one or more test suites and corresponding test cases, using his/her favorite tool/IDE 
  • Each test case could be linked to the corresponding requirement/user story in Jira by adding its key as a tag
  • Tests could then be run locally, or from the CI pipeline
  • Unique, non-duplicating, Test entities would be auto-provisioned in Xray, corresponding to each test case; tester could also, optionally, enforce the result to an existing Test entity by specifying its issue key as a tag


Let’s take the following .robot file as an example, which acts as a suite containing one test case.

Code Block
titlelogin_tests/valid_login.robot
collapsetrue
*** Settings ***
Documentation     A test suite with a single test for valid login.
...
...               This test has a workflow that is created using keywords in
...               the imported resource file.
Resource          resource.robot

*** Test Cases ***
Valid Login
    [Tags]  CALCROB-111  CALC-2UI
    Open Browser To Login Page
    Input Username    admindemo
    Input Password    adminmode
    Submit Credentials
    Welcome Page Should Be Open
    [Teardown]    Close Browser

...



The previous Robot files file use a common resource that contains some generic variables and some reusable "keywords" (i.e., steps).
 Code Block
titlelogin_tests/resource.robot
collapsetrue
*** Settings ***
Documentation     A resource file with reusable keywords and variables.
...
...               The system specific keywords created here form our own
...               domain specific language. They utilize keywords provided
...               by the imported Selenium2LibrarySeleniumLibrary.
Library         BuiltIn
Library         Selenium2LibrarySeleniumLibrary

*** Variables ***
${SERVER}         localhost:8080192.168.56.1:7272
${BROWSER}        chromeFirefox
${DELAY}          0.5
${VALID USER}       admindemo
${VALID PASSWORD}   admin mode
${LOGIN URL}      http://${SERVER}/login.jsp
${WELCOME URL}    http://${SERVER}/welcome.html
${ERROR URL}/secure/Dashboard.jspa      http://${SERVER}/error.html

*** Keywords ***
Open Browser To Login Page
    Open Browser    ${LOGIN URL}    ${BROWSER}
    Maximize Browser Window
    Set Selenium Speed    ${DELAY}
    Login Page Should Be Open

Login Page Should Be Open
    Title Should Be    Log in - Xray Demo Environment (JIRA 7)Login Page

Go To Login Page
    Go To    ${LOGIN URL}
    Run Keyword And Ignore Error        Get Alert Message
    Login Page Should Be Open

Input Username
    [Arguments]    ${username}
    Input Text    login-form-username_field    ${username}

Input Password
    [Arguments]    ${password}
    Input Text    login-form-password_field    ${password}

Submit Credentials
    Click Button    login-form-submit_button

Welcome Page Should Be Open
    Location Should Be    ${WELCOME URL}
    Title Should Be    Tests Dashboard - Xray Demo Environment (JIRA 7)Welcome Page

A Test issue will be auto-provisioned the first time you import the results, based on the name of the test case and of the corresponding test suites.
If you maintain the test case name and the respective test suites, the Test will be reused on subsequent result imports. You may always enforce the results to be reported against an existing Test, if you wish so: just specify its issue key as a tag.
Tags can also be used to cover an existing requirement/user story (e.g. “ROB-11”): when a requirement issue key is given, a link between the test and the requirement is created during the results import process. 
Otherwise, tags are mapped as labels on the corresponding Test issue.

Image Added


Importing results is as easy as submitting them to the REST API with a POST request (e.g. curl), or by using one of the CI plugins available for free (e.g. Xray Jenkins plugin).

Image Added

In Jira, using Xray, the team can look at the Test Run details which include the overall result and also specifics about each keyword, including duration and status.

Image Added

We can all see that a Test case/issue was auto-provisioned based on the name of the test case and of the corresponding test suites.








After running the tests and generating the Robot XML report (e.g., output.xml), it can be imported to Xray via the REST API.
...
You will see information about each Robot keyword (i.e., step) and its corresponding status in the Context section of the Execution Details of the Generic Test.


Tracking it everywhere
On the user story issue screen
References
Right from within the user story issue screen, we now see one test (i.e. automated script) covering it. We can also see its latest result and how it impacts the overall coverage calculation for the user story; if the user story shows as “OK”, you know that all tests covering it passed, accordingly with the latest results obtained for each one of them.

Image Added
On the Agile Board
On Agile Boards (e.g. Scrum boards), we can now assess the coverage of our user story taking into account the testing results.
We may also track the overall Test Plan consolidated progress on the Test Plan issue related card. Note that we could include Test Executions in the board if we wish so; however, in CI scenarios that could be counterproductive.

Image Added
On the Test Plan
At the Test Plan-level, the entity that defines the scope of testing and tracks its progress, we can quickly assess the latest consolidated test results (i.e. the latest result obtained for each Test being tracked).
Image Added
References