About the Robot Framework
The Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD).
It has easy-to-use tabular test data syntax and it uses the keyword-driven testing approach.
Robot Framework Concepts
The Robot Framework modular architecture can be extended with bundled and self-made test libraries.
A test suite consists of a file containing test cases. A directory containing these suite files creates a nested structure of test suites.
Robot is capable of generating a report for each execution in its specific XML format (xUnit) as well as HTML data formats.
The native Robot Framework's XML format has a well-defined schema and represents the pattern in which the framework executes the keywords given in that particular test suite.
Importing Robot Framework XML reports
Each Robot test case can be marked with a Tag identifying the requirement being tested; it also can be a Test definition already present in Jira. Tagging a requirement in a test case will create a link between the test case and the requirement. Identifying a test case with an issue key Tag, will create a test execution for the pre-existing Test (it does not create any new Test in this case).
It is important that the "Requirements issue types mapping" is configured at the point of import; otherwise, no link will be created. Refer to Issue Type Mapping for more information.
Below is a simplified example of a Robot Framework XML report. There are some nested test suites, and one failed test marked with Tags. The WEB-1 tag corresponds to an existing Story we want to test, while WEB-3 is the Test issue key which contains the test definition.
Robot Framework v4.0 XML reports
As from 5.0.0, Xray supports output format from Robot Framework 4.0, including new features such as the SKIP status, native IF/ELSE and nested control structures.
Results in Xray Test Execution Page
Each Robot test case is mapped to a Generic Test in Jira, having the summary with the name of the test case, and the Generic Test Definition field contains the concatenated names of the test suites, along with the name of the test case. Note that the Robot Framework considers the base folder of the project as the first test suite.
The way you run your tests also affects Robot's XML, so if you execute the file from somewhere else or you directly execute the file by passing it as an argument, the test suite's information will potentially be different.
The name, error message, duration and status of Robot's keywords will be imported and stored in the test run results.
Information about each Robot keyword (i.e., step), along with the corresponding status, is displayed in the Context section of the Execution Details of the Generic Test.
The example below shows the execution details page of a Test containing both test keywords as well as the final test teardown keyword.
Robot Framework's test case status maps directly to Xray test run status FAIL and PASS.
For passed keywords, the message will be imported from the inner <msg> element within the respective <kw> element (only if the <msg> element has the attribute "level" having one these values: "INFO", "WARN", "ERROR", "FAIL").
For failed keywords, the message will be imported from the inner <msg> element within the respective <kw> element. If the <msg> element does not exist, it will be imported from the text associated with the <status> element defined within the <test> element.
This information will be shown in the execution screen of the Test Run.
Notes and Limitations
- attachments (e.g. screenshots and other files) are not supported/imported as they are not embedded in the XML report; it seems that some libraries add references to their local paths in the <doc/> element but these cannot be imported as they are external to the report
- nested keywords are not supported