Introduction
The Robot Framework is a generic Test automation framework for acceptance Testing and acceptance Test-driven development (ATDD).
It has easy-to-use tabular test data syntax and uses the keyword-driven testing approach.
Concepts
The Robot Framework modular architecture can be extended with bundled and custom Test libraries (Figure 1):
- A Test suite consists of a file containing Test cases.
- A directory containing these suite files creates a nested structure of test suites.
Robot is capable of generating a report for each execution in its specific XML format (xUnit) as well as HTML data formats.
The native Robot Framework XML format has a well-defined schema that represents the pattern in which the framework executes the keywords given in that particular Test suite.
Operations
Importing Robot Framework XML Reports
Each Robot Test case can be marked with a tag identifying the requirement being tested; it can also be a Test definition already present in Jira. Tagging a requirement in a Test case will create a link between the Test case and the requirement. Identifying a Test case with an Issue key tag will create a Test execution for the pre-existing Test (it does not create any new Test in this case).
It is important that the Requirements Issue types mapping is configured at the point of import; otherwise, no link will be created.
Below is a simplified example of a Robot Framework XML report. There are some nested Test suites, and one failed test marked with tags. The WEB-1 tag corresponds to an existing story you want to test, while WEB-3 is the Test Issue key which contains the Test definition.
Robot Framework v4.0 XML Reports
Starting from version 5.0.0, Xray supports output format from Robot Framework 4.0, including new features such as the SKIP status, native IF/ELSE, and nested control structures.
<?xml version="1.0" encoding="UTF-8"?>
<robot generator="Robot 4.0.1 (Python 3.9.5 on darwin)" generated="20210511 14:11:38.099" rpa="false" schemaversion="2">
<suite id="s1" name="Demo" source="/Users/dpca/Downloads/robot40_outputxml_changes/demo.robot">
<test id="s1-t1" name="Example using conditional statements">
<if>
<branch type="IF" condition="${var1} > 0">
<kw name="Sleep" library="BuiltIn">
<arg>1</arg>
<doc>Pauses the test executed for the given time.</doc>
<status status="NOT RUN" starttime="20210511 14:11:38.144" endtime="20210511 14:11:38.144"/>
</kw>
<status status="NOT RUN" starttime="20210511 14:11:38.144" endtime="20210511 14:11:38.144"/>
</branch>
<branch type="ELSE">
<kw name="Sleep" library="BuiltIn">
<arg>2</arg>
<doc>Pauses the test executed for the given time.</doc>
<msg timestamp="20210511 14:11:40.146" level="INFO">Slept 2 seconds</msg>
<status status="PASS" starttime="20210511 14:11:38.144" endtime="20210511 14:11:40.146"/>
</kw>
<status status="PASS" starttime="20210511 14:11:38.144" endtime="20210511 14:11:40.146"/>
</branch>
<status status="PASS" starttime="20210511 14:11:38.144" endtime="20210511 14:11:40.146"/>
</if>
<status status="PASS" starttime="20210511 14:11:38.143" endtime="20210511 14:11:40.146"/>
</test>
<test id="s1-t2" name="Example using nested for loops">
<for flavor="IN">
<var>${robot}</var>
<value>@{ROBOTS}</value>
<iter>
<var name="${robot}">Cortana</var>
<for flavor="IN">
<var>${dog}</var>
<value>@{DOGS}</value>
<iter>
<var name="${dog}">James</var>
<kw name="Log" library="BuiltIn">
<arg>${robot} ${dog}</arg>
<doc>Logs the given message with the given level.</doc>
<msg timestamp="20210511 14:11:40.148" level="INFO">Cortana James</msg>
<status status="PASS" starttime="20210511 14:11:40.148" endtime="20210511 14:11:40.148"/>
</kw>
<status status="PASS" starttime="20210511 14:11:40.148" endtime="20210511 14:11:40.148"/>
</iter>
<iter>
<var name="${dog}">Paul</var>
<kw name="Log" library="BuiltIn">
<arg>${robot} ${dog}</arg>
<doc>Logs the given message with the given level.</doc>
<msg timestamp="20210511 14:11:40.149" level="INFO">Cortana Paul</msg>
<status status="PASS" starttime="20210511 14:11:40.148" endtime="20210511 14:11:40.149"/>
</kw>
<status status="PASS" starttime="20210511 14:11:40.148" endtime="20210511 14:11:40.149"/>
</iter>
<status status="PASS" starttime="20210511 14:11:40.147" endtime="20210511 14:11:40.149"/>
</for>
<status status="PASS" starttime="20210511 14:11:40.147" endtime="20210511 14:11:40.149"/>
</iter>
<iter>
<var name="${robot}">Terminator</var>
<for flavor="IN">
<var>${dog}</var>
<value>@{DOGS}</value>
<iter>
<var name="${dog}">James</var>
<kw name="Log" library="BuiltIn">
<arg>${robot} ${dog}</arg>
<doc>Logs the given message with the given level.</doc>
<msg timestamp="20210511 14:11:40.150" level="INFO">Terminator James</msg>
<status status="PASS" starttime="20210511 14:11:40.149" endtime="20210511 14:11:40.150"/>
</kw>
<status status="PASS" starttime="20210511 14:11:40.149" endtime="20210511 14:11:40.150"/>
</iter>
<iter>
<var name="${dog}">Paul</var>
<kw name="Log" library="BuiltIn">
<arg>${robot} ${dog}</arg>
<doc>Logs the given message with the given level.</doc>
<msg timestamp="20210511 14:11:40.150" level="INFO">Terminator Paul</msg>
<status status="PASS" starttime="20210511 14:11:40.150" endtime="20210511 14:11:40.150"/>
</kw>
<status status="PASS" starttime="20210511 14:11:40.150" endtime="20210511 14:11:40.150"/>
</iter>
<status status="PASS" starttime="20210511 14:11:40.149" endtime="20210511 14:11:40.150"/>
</for>
<status status="PASS" starttime="20210511 14:11:40.149" endtime="20210511 14:11:40.150"/>
</iter>
<iter>
<var name="${robot}">AI</var>
<for flavor="IN">
<var>${dog}</var>
<value>@{DOGS}</value>
<iter>
<var name="${dog}">James</var>
<kw name="Log" library="BuiltIn">
<arg>${robot} ${dog}</arg>
<doc>Logs the given message with the given level.</doc>
<msg timestamp="20210511 14:11:40.151" level="INFO">AI James</msg>
<status status="PASS" starttime="20210511 14:11:40.151" endtime="20210511 14:11:40.151"/>
</kw>
<status status="PASS" starttime="20210511 14:11:40.150" endtime="20210511 14:11:40.151"/>
</iter>
<iter>
<var name="${dog}">Paul</var>
<kw name="Log" library="BuiltIn">
<arg>${robot} ${dog}</arg>
<doc>Logs the given message with the given level.</doc>
<msg timestamp="20210511 14:11:40.151" level="INFO">AI Paul</msg>
<status status="PASS" starttime="20210511 14:11:40.151" endtime="20210511 14:11:40.151"/>
</kw>
<status status="PASS" starttime="20210511 14:11:40.151" endtime="20210511 14:11:40.151"/>
</iter>
<status status="PASS" starttime="20210511 14:11:40.150" endtime="20210511 14:11:40.151"/>
</for>
<status status="PASS" starttime="20210511 14:11:40.150" endtime="20210511 14:11:40.151"/>
</iter>
<status status="PASS" starttime="20210511 14:11:40.147" endtime="20210511 14:11:40.151"/>
</for>
<status status="PASS" starttime="20210511 14:11:40.147" endtime="20210511 14:11:40.152"/>
</test>
<test id="s1-t3" name="This test will be skipped explictly">
<kw name="Skip" library="BuiltIn">
<doc>Skips the rest of the current test.</doc>
<msg timestamp="20210511 14:11:40.152" level="SKIP">Skipped with Skip keyword.</msg>
<status status="SKIP" starttime="20210511 14:11:40.152" endtime="20210511 14:11:40.153"/>
</kw>
<status status="SKIP" starttime="20210511 14:11:40.152" endtime="20210511 14:11:40.153">Skipped with Skip keyword.</status>
</test>
<test id="s1-t4" name="This test will always pass">
<kw name="Pass Execution" library="BuiltIn">
<arg>all good here</arg>
<doc>Skips rest of the current test, setup, or teardown with PASS status.</doc>
<msg timestamp="20210511 14:11:40.153" level="INFO">Execution passed with message:
all good here</msg>
<status status="PASS" starttime="20210511 14:11:40.153" endtime="20210511 14:11:40.154"/>
</kw>
<status status="PASS" starttime="20210511 14:11:40.153" endtime="20210511 14:11:40.154">all good here</status>
</test>
<test id="s1-t5" name="This test will always fail">
<kw name="Fail" library="BuiltIn">
<arg>failed on purpose</arg>
<doc>Fails the test with the given message and optionally alters its tags.</doc>
<msg timestamp="20210511 14:11:40.154" level="FAIL">failed on purpose</msg>
<status status="FAIL" starttime="20210511 14:11:40.154" endtime="20210511 14:11:40.154"/>
</kw>
<status status="FAIL" starttime="20210511 14:11:40.154" endtime="20210511 14:11:40.155">failed on purpose</status>
</test>
<test id="s1-t6" name="This test will also fail but will report as skip">
<kw name="Fail" library="BuiltIn">
<arg>failed also on purpose</arg>
<doc>Fails the test with the given message and optionally alters its tags.</doc>
<msg timestamp="20210511 14:11:40.156" level="FAIL">failed also on purpose</msg>
<status status="FAIL" starttime="20210511 14:11:40.155" endtime="20210511 14:11:40.156"/>
</kw>
<tag>skipme</tag>
<status status="SKIP" starttime="20210511 14:11:40.155" endtime="20210511 14:11:40.156">Test failed but its tags matched '--SkipOnFailure' and it was marked skipped.
Original failure:
failed also on purpose</status>
</test>
<test id="s1-t7" name="This test will fail due to the first kw, while 2nd one is marked as not run">
<kw name="Fail" library="BuiltIn">
<arg>failed on purpose</arg>
<doc>Fails the test with the given message and optionally alters its tags.</doc>
<msg timestamp="20210511 14:11:40.157" level="FAIL">failed on purpose</msg>
<status status="FAIL" starttime="20210511 14:11:40.157" endtime="20210511 14:11:40.157"/>
</kw>
<kw name="Sleep" library="BuiltIn">
<arg>1</arg>
<doc>Pauses the test executed for the given time.</doc>
<status status="NOT RUN" starttime="20210511 14:11:40.157" endtime="20210511 14:11:40.157"/>
</kw>
<status status="FAIL" starttime="20210511 14:11:40.156" endtime="20210511 14:11:40.157">failed on purpose</status>
</test>
<status status="FAIL" starttime="20210511 14:11:38.103" endtime="20210511 14:11:40.158"/>
</suite>
<statistics>
<total>
<stat pass="3" fail="2" skip="2">All Tests</stat>
</total>
<tag>
<stat pass="0" fail="0" skip="1">skipme</stat>
</tag>
<suite>
<stat pass="3" fail="2" skip="2" id="s1" name="Demo">Demo</stat>
</suite>
</statistics>
<errors>
</errors>
</robot>
Results in Xray Test Execution Page
Xray Integration with Robot Framework 5.0, 6.0 and 7.0
Starting from version 7.8.0, Xray fully supports the output format from Robot Framework 5.0, 6.0, and 7.0. These updates bring a range of new features and enhancements to streamline and optimize your test automation processes.
Key Features Implemented in Robot Framework 5.0
TRY/EXCEPT Syntax
- Description: handles exceptions within your Test cases using TRY/EXCEPT blocks.
- Benefits: allows for graceful error handling during Test execution, ensuring robust and reliable Test scripts.
WHILE Loops
- Description: introduces WHILE loops, enabling the repetition of a set of steps while a condition is true.
- Benefits: useful for dynamic scenarios where iterative execution is required until a specific condition is met.
- Inline IF (Conditional Execution)
- Description: utilizes inline IF statements to conditionally execute keywords based on a condition.
- Benefits: simplifies Test logic, making scripts more readable and maintainable.
Flow Control Keywords (RETURN, BREAK, and CONTINUE)
- Description: controls the flow of execution within loops using new keywords.
- RETURN: exits the current keyword.
- BREAK: stops the loop.
- CONTINUE: skips the current iteration and proceeds to the next one.
- Benefits: provides greater control over test execution flow, allowing for more flexible and efficient test scripts.
- Description: controls the flow of execution within loops using new keywords.
Results in Xray Test Execution Page
Key Features Implemented in Robot Framework 6.0
Error Tag
- Description: automatically adds an
Errortag to Test cases that fail due to errors (as opposed to failures due to assertion errors). - Benefits: helps distinguish between different types of Test failures, making it easier to identify and address Issues.
Results in Xray Test Execution Page
Key Features Implemented in Robot Framework 7.0
Native VAR Syntax for Variables
- Description: Robot Framework supports a native syntax for creating variables using the VAR keyword.
- Benefits: simplifies variable creation and improves readability.
Results in Xray Test Execution Page
By leveraging these new features in Robot Framework 5.0, 6.0, and 7.0, you can enhance the flexibility, readability, and robustness of your automated Test scripts. Xray's support for these output formats ensures seamless integration and efficient Test management within your projects.
Entities
Each Robot test case is mapped to a Generic Test in Jira, with the summary containing the name of the test case. The Generic Test Definition field includes the concatenated names of the test suites along with the name of the test case. Note that Robot Framework considers the base folder of the project as the first test suite.
The way you run your tests affects Robot's XML. Executing the file from different locations or passing it as an argument directly may result in different test suite information.
The text will be trimmed to match the 255-character limit on the summary field.
The name, error message, duration, and status of Robot's keywords will be imported and stored in the Test run results.
The test case keywords and the test case setup/teardown keywords are imported individually, allowing you to see their respective results. However, the test suite setup/teardown keywords are not imported.
Information about each Robot keyword (i.e., step), along with the corresponding status, is displayed in the Context section of the Execution Details of the Generic Test. The example below (Figure 8) shows the execution details page of a test containing both test keywords and the final test teardown keyword.
Status
Robot Framework's test case status maps directly to Xray test run statuses: FAIL and PASS.
Message
For passed keywords, the message will be imported from the inner <msg> element within the respective <kw> element (only if the <msg> element has the attribute "level" with values "INFO", "WARN", "ERROR", or "FAIL". "WARN", "ERROR", "FAIL").
For failed keywords, the message will be imported from the inner <msg> element within the respective <kw> element. If the <msg> element does not exist, it will be imported from the text associated with the <status> element defined within the <test> element.
This information will be shown on the Execution screen of the Test run.
Working example
Notes and Limitations
- Attachments (e.g., screenshots and other files) are not supported or imported as they are not embedded in the XML report. Some libraries add references to their local paths in the
<doc/>element, but these cannot be imported as they are external to the report. - Nested keywords are not supported.