What you'll learn
- Brief introduction to Robot Framework
- Creation of Robot scripts in Test Case Designer
- Comparison of coverage and execution metrics
In the "How to Optimize Data-driven Automation with Test Case Designer" tutorial (Server/DC ; Cloud), we made a quick note that the efficient script-based approach is also possible in Test Case Designer (TCD). So, we wanted to explore that topic more and give the spotlight to integrating TCD with Robot Framework.
Robot Framework is an open source test automation framework for acceptance testing and acceptance test-driven development. It follows different test case styles − keyword-driven, behaviour-driven and data-driven − for writing test cases.
Some of the key Robot Framework advantages include ease of installation and use, good support for both external libraries and built-in/custom keywords, and more readable & maintainable test cases.
Overview
One of the most common concerns we hear from clients is:
“Our current tests are good enough. Why invest the time and resources in adopting the Test Case Designer methodology if it doesn’t move the needle?”
That may be true, but we don’t have to guess - TCD Analysis capabilities allow us to make a pretty accurate comparison (you can learn more in the "How Are TCD Tests Objectively Superior?" tutorial from the "Why is Test Case Designer helpful?" section). For this tutorial, we will use 2 test suites created for the same requirement/coverage goal:
- "TCDBookingTests", representing the optimized output from TCD with 15 tests;
- "TypicalBookingTests", representing the manually designed suite with 33 tests (based on our experience, it shows an example of a "pretty good job" for the manual effort type).
The requirement is to provide hotel availability options for the given reservation details at https://www.trip.com/. We check this by validating that clicking the "Search" button successfully leads to the "Results" page (by "properties found" page content).
We will first describe the integration steps for "TCDBookingTests" and then compare the coverage and execution metrics between the two.
Prerequisites
In order to run this tutorial you need to have Python and Robot Framework installed.
Integrating Test Case Designer with Robot Framework
Build a TCD Model
It is built using a fairly straightforward approach of a parameter per UI field with a couple things to note:
- For the Destination field, we are more interested in classes of search terms, less in the syntax rules. So, we use 2 Values to represent categories, then add specific search terms using Value Expansions.
- The date-related parameters will need to be updated within the model to maintain execution accuracy. The screenshot represents the model state at the time of publication. Alternatively, you can consider using abstract values in TCD ("this month", "next month", "last day of the month", etc.), then let automation determine the specifics.
- For conditional steps, Robot Framework allows to use any syntax as the trigger, so we can adjust the value naming to match the context - "checks" for the checkbox, numbers for the rating stars. Neither wording has any special function within TCD itself.
Constraints should account for the date logic and the room-vs-people limitations:
For scenario generation, we assume the exact checkout date and the ratings are not as important, so we switch them to 1-way:
You can import the model and try this process yourselves:
Create a Robot script
The key advantage at this step is that a user creates & maintains a single data-driven template in TCD Automate and exports as many tests as there are rows on the TCD Scenarios screen, with all the parameter values replaced inline. That combines efficiency and clarity/ease of use.
Settings and Variables are defined similarly to any other Robot script. In the Test Cases section, there are a couple of points to highlight:
*** Test Cases ***
Valid Booking Requests_TC<Test Case>_for <destination> with <rooms> rooms and <adults> adults
[Tags] TAC-242
Given browser is opened to start page
When user selects destination "<destination>"
And user selects May 2023 "<checkindate>" check-in date and "<checkoutmonth>" "<checkoutdate>" check-out date
And user selects "<rooms>" rooms for "<adults>" adults and "<children>" children
And user "<workcheckbox>" work checkbox and selects "<rating>" star rating
Then clicking "Search" button generates results successfully
- TCD only supports the Gherkin style of Robot scripting.
- Requirement linking is handled via Tags.
- As in other TCD Scripting examples, the steps are parameterized in order to connect to the Scenarios table (e.g. "<destination>").
In the Keywords section, there are typically 3 ways of handling TCD parameters:
Direct argument
user selects destination "${destination}" Input Text ${DESTINATION TXTFIELD} ${destination}Part of xpath
Click Element //div[@class = 'c-calendar-month__title' and text() = '${checkoutmonth}']/following-sibling::div//span[@class = 'day' and text() = '${checkoutdate}']Part of condition (in this example, the age of children ("12") was not important enough to be a parameter/value expansion, so it's hardcoded directly in the script)
Run Keyword If '${children}' == '1' Click Element ${CHILDRENPLUS BTN} Run Keyword If '${children}' == '1' Select From List By Value ${CHILD1AGE DROPDOWN} 12
Once the script is ready, we will export it into the Robot format:
You can choose to leverage your own IDE for the template writing (to benefit from IntelliSense, etc.) then copy it into TCD for the export process.
Execute and import results
We can run the tests from the command line, specifying the report name and format:
robot -d reports --output TCDDemoReport.xml TCDBookingTests.robot
If we choose to run them in parallel, we can install Pabot and utilize the following command:
pabot --testlevelsplit -d reports --output TCDDemoReport.xml TCDBookingTests.robot
Once the script completes the execution, we can import results using your CI/CD tool of choice, eventually with one of available plugins for them, or invoking Xray's REST API directly using `curl` utility. For example (parts in {} should be customized based on your details):
curl -H "Content-Type: multipart/form-data" -u {username}:{password} -F "file=@reports/{ReportName}.xml" “https://{JiraURL}/rest/raven/1.0/import/execution/junit?projectKey={ProjectKey}&testPlanKey={TestPlanIssueKey}”
CI/CD
For the CI/CD process, please refer to this collection of tutorials - DC/Server ; Cloud
Also, you can learn more about Robot Framework integrations from these articles:
- Taking advantage of Robot XML reports (DC/Server)
- Testing using Robot Framework integration in Python or Java (DC/Server)
As a result, you should see 15 Test issues (linked to the requirement story and to the Test Plan issue) and a Test Execution issue (with the step-by-step breakdown under Execution details of each run):
One scenario in "TCDBookingTests" fails because the UI error tooltip appears about the maximum stay duration of 31 days. Before the selections, that limitation doesn't seem to be noted anywhere.
This could be an example of the execution feedback loop for combinatorial test design: we will often exercise the combinations that haven't been explicitly mentioned in requirements. If the 31-day limit hasn't been caught during the collaborative model creation, then, based on the execution failure, we would confirm the expected behavior and, if the error is valid,
- add the constraint to the TCD model between checkindate[15,24] and checkoutdate[30, 31] and/or
- replace the value lists to focus more on the valid range.
A few scenarios in "TypicalBookingTests" fail for the same reason.
Comparison and Conclusion
Here are the final versions of both suites:
TCDBookingTests.robot TypicalBookingTests.robot
Based on the corresponding execution reports, we can compare the time:
~15 minutes for the TCD suite vs ~27 minutes for the manually designed suite without parallelization (~40 seconds difference with parallelization). This doesn't account for the much faster time to create the TCD test suite.
With the help of TCD analysis capabilities, we can compare interaction coverage - 100% for the TCD suite vs 84.9% for the manually designed suite after 15 tests (96.8% for the manually designed suite after all 33 tests).
And, while harder to estimate, the increase in script maintenance efficiency (1 data-driven template in TCD vs individual scripts in typical approach) is important to keep in mind.
Coverage graph for the "Optimized" suite:
Coverage graph for the "Typical" suite. Notice that the horizontal axis:
1) is on a different scale, which makes the shape look more accelerated than the one above (while the actual percentages are lower on a test-by-test basis);
2) goes beyond 33 as TCD has to complete the mixed-strength coverage goal.
Tips
- As mentioned above, this tutorial demonstrates the approach where "complete" Robot tests in Gherkin style are generated from TCD. A data-driven implementation similar to the Playwright/Cypress tutorial is also possible via a CSV/pipe-delimited data table exported from the TCD Scenarios screen + the Test Template in RF (or using a specialized library).
- For simplicity of the example, all executable code is in 1 file. You can split it into Resource.robot and Booking.robot, if desired.
- If the desired algorithm goal is 2-way, not mixed-strength, then Coverage Matrix can be used to review the specific pairwise gaps in addition to the aggregate coverage percentage.