Page History
...
All iterations for a given test are executed within the context of the same test run. Each iteration can be expanded and the steps executed individually. The step parameters will be replaced by the corresponding iteration values. The steps affect the iteration status that, in turn, affects the overall test run status.
TODO: link to execution page
Please check the Execution page for more details.
Parameterized tests vs test environments
Although test environments can be thought of as a particular case
...
of parameterized tests
...
(given that environments can be considered parameters of test cases), datasets
...
might not
...
be the best choice for environmental variables that are not embedded in the test specification (e.g. manual test steps).
As such, it is not good practice to use datasets with environment variables such as Browsers, OSs, Databases, or Devices. This is because:
- Test engineers often configure a specific setup (or environment) to execute a set of tests. In this case, test engineers might want to execute the tests oriented to the environment and not the test case. This means they execute all the test cases first for one environment, and then for another one, and so on. This way, having all these environments within the same test run is not ideal as users would have to jump from one test run to the other without finishing the execution.
- Usually, test environments are independent variables or dimensions. For instance, let's consider the following test environment: Browser, Database, and OS. If a test case fails only in a specific browser, once the bug is fixed, we might not want to re-execute the same test for all databases and OS's as we are confident that the change did not affect these variables. With datasets, if one
...
- iteration fails, you need to re-execute all
...
- the iterations again (if you use a different test run, of course).
- Xray does not provide reports based on parameters.
In conclusion, if your parameters are environment variables that do not need to be included in the test specification, don't use datasets. Use test environments instead.
| Info | ||
|---|---|---|
| ||
We have plans to improve how test environments are managed in Xray. It will be possible to specify or generate combinations of environments for different variables (just like with combinatorial parameters) and have a central place to manage test environments. This feature also includes dedicated reports that are oriented for test environments.TODO: link to Roadmap. Please check our roadmap. |
Defining a parameterized test
...