Test parameterization is a powerful practice that allows the same test to be executed multiple times with different parameters. Parameters are similar to input values (variables) that can change with each execution.
Parameterized tests in Xray are defined just like any other test with the addition of some parameter names within the specification using the following notation: ${PARAMETER_NAME}. This notation is used to reference parameters within thetest steps.
Preconditionissues can also be parameterized by including parameter names in the precondition specification.
The parameters, along with their values, are defined within adataset. A dataset is a collection of data represented with a tabular view where every column of the table represents a particular variable (orparameter), and each row corresponds to a given record (oriteration) of the dataset. The number of rows in the dataset determines the number of iterations to execute.
A dataset can be defined in the following entities/scopes:
Test (default dataset)
Test Plan - Test
Test Execution - Test (Test Run)
The closest dataset to the test run will be the one used to generate the iterations, effectively overriding any dataset defined in higher levels.
All iterations for a given test are executed within the context of the same test run. Each iteration can be expanded, and the steps executed individually. The step parameters will be replaced by the corresponding iteration values. The steps affect the iteration status that, in turn, affects the overall test run status.
Modular test design is a way of promoting test case reusability and composition across a large test repository. To design modular tests, you can create a test where some of the stepscallor include other test cases. This prevents testers from having to write the same steps over and over again in different high-level tests. Using a modular design approach, any test can become a building block of more extensive test scenarios. Still, they can also be executed individually if needed.
A called test can, in turn, also call other tests. You can compose a test scenario with up to five levels of depth.
Modular tests can also be parameterized. When calling a test, you can provide new parameter values according to the parent test data.
Upon executing, Xray will unfold all called test steps in the test run. This becomes transparent to testers as they only have to follow and execute the steps on the execution, even though test steps might come from different tests issues.
A common use case for modular tests is end-to-end testing. End-to-end tests often need to pass through the same area or component of the application before asserting the final result. With modular test design, you can reuse the tests for these common areas or components.
TheAutomated Steps Library will be a project-level steps library for BDD Tests/Preconditions, containing all the Gherkin steps referred by these issues.
Thus, it provides a facility to store and manage all steps, encouraging users to easily reuse steps across Tests/Preconditions with step autocompletion and discoverability when editing BDD scenarios and backgrounds with Xray.
Expand
title
Time Tracking
Panel
borderColor
#EAECEE
borderWidth
1px
borderStyle
solid
Right now, a Test Run contains the start and finish date which are set automatically, although users can also change the start date manually. However, users can not stop and start the execution as needed and the total time spent on the Test Run (finish - start) might not reflect the time spent by the user executing the Test.
We plan to provide a feature to allow users to pause and continue with the execution directly on the Test Run screen.
Ability to call an existing Test from the step of another Test and thus use tests as reusable building blocks for the composition of more complex testing scenarios.
The called test can be executed independently or be used as a step of a broader test (e.g. "login as the admin user").
Wish list for future versions
...
Expand
title
Testing board
Panel
borderColor
#EAECEE
borderWidth
1px
borderStyle
solid
The goal behind the testing board is to provide a centralized hub for accessing Xray entities and activities in order to improve navigation and discoverability. For now, this board features the Test repository, the Test Plan board, and project reports.
The testing board will be improved to include other Xray entities such as Preconditions, Test Sets, and Test Executions. The Test Run and execution screen will have the testing board in context so that users can navigate to any other test activity.
Expand
title
Automated Steps Library
Panel
borderColor
#EAECEE
borderWidth
1px
borderStyle
solid
TheAutomated Steps Library will be a project-level steps library for BDD Tests/Preconditions, containing all the Gherkin steps referred by these issues.
Thus, it provides a facility to store and manage all steps, encouraging users to easily reuse steps across Tests/Preconditions with step autocompletion and discoverability when editing BDD scenarios and backgrounds with Xray.
Expand
title
Test Environment variables
Panel
borderColor
#EAECEE
borderWidth
1px
borderStyle
solid
This feature will allow test engineers to create or generate environments with variables such as Browser, Operating System, Database, etc. These environments (or configurations) can be managed and assigned to Test Plans and Test Plan folders. Test Executions can then be created automatically for each environment/configuration.
Xray will provide a report to track the progress of Test Runs by environment in the context of a specific Test Plan.
...
Expand
title
Test Executions Metrics report
Panel
borderColor
#EAECEE
borderWidth
1px
borderStyle
solid
This report will feature a list of Test Execution issues along with metrics. Possible usage scenarios:
analyze both the progress of the Test Executionand the success rate (i.e., the % of Tests contributing to the requirement's OK status)
see the number of manual Tests vs. others in the Test Execution
see the overall execution status (i.e., the current status of the Test Runs)
see the number of opened/closed linked defects, in the context of the Test Execution
Right now, a Test Run contains the start and finish date which are set automatically, although users can also change the start date manually.However, users can not stop and start the execution as needed and the total time spent on the Test Run (finish - start) might not reflect the time spent by the user executing the Test.We plan to provide a feature to allow users to pause and continue with the execution directly on the Test Run screen.
Expand
title
Daily history coverage report
Panel
borderColor
#EAECEE
borderWidth
1px
borderStyle
solid
This report will show a daily historical view of Requirements coverage. Users will be able to analyze the evolution of the Requirement's coverage status over time for a particular analysis scope: Latest, Version or Test Plan (for each Environment). This way they can estimate if they are going to keep the planed released date according to the coverage status trend.
