Page History
...
All iterations for a given test are executed within the context of the same test run. Each iteration can be expanded and the steps executed individually. The step parameters will be replaced by the corresponding iteration values. The steps affect the iteration status that, in turn, affects the overall test run status.
Please check the Execution page for more details.
Parameterized tests vs test environments
Although test environments can be thought of as a particular case of parameterized tests (given that environments can be considered parameters of test cases), datasets might not be the best choice for environmental variables that are not embedded in the test specification (e.g. manual test steps).
As such, it is not good practice to use datasets with environment variables such as Browsers, OSs, Databases, or Devices. This is because:
- Test engineers often configure a specific setup (or environment) to execute a set of tests. In this case, test engineers might want to execute the tests oriented to the environment and not the test case. This means they execute all the test cases first for one environment, and then for another one, and so on. This way, having all these environments within the same test run is not ideal as users would have to jump from one test run to the other without finishing the execution.
- Usually, test environments are independent variables or dimensions. For instance, let's consider the following test environment: Browser, Database, and OS. If a test case fails only in a specific browser, once the bug is fixed, we might not want to re-execute the same test for all databases and OS's as we are confident that the change did not affect these variables. With datasets, if one iteration fails, you need to re-execute all the iterations again (if you use a different test run, of course).
- Xray does not provide reports based on parameters.
In conclusion, if your parameters are environment variables that do not need to be included in the test specification, don't use datasets. Use test environments instead.
| Info | ||
|---|---|---|
| ||
We have plans to improve how test environments are managed in Xray. It will be possible to specify or generate combinations of environments for different variables. Please check our roadmap. |
...
7.2. Click on the "Generate all combinations." A confirmation dialog will appear. Confirming the changes will apply the cartesian product between the combinational parameter values and the seeding parameter rows.
8.
...
Move left/right:
You can re-order any parameters in your dataset, by selecting the "Move left/right" option.
8.1. Open the options menu in either a Combinatorial or non-combinatorial (seeding) parameter, you will have a "Move left/right" option available. Notice that if the parameter is already place on the most outer left (right) position, the "Move left" (right) option is not available.
8.2. Select the move left/right option, This will swap the positions of the parameters.
9. After having performed all changes in the dataset, you need to press the "Save" button in order to persist the dataset into the database.
Importing a dataset from a CSV file
Besides defining a dataset by creating parameters and setting their values directly using the Xray UI, it is also possible to import an existing dataset from a CSV file.
In order to import a CSV file:
1. Open the dataset dialog.
2. Choose the "Import from CSV" option available on the Import dropdown button (located on the empty state view or on the dataset main button toolbar). A dialog will appear.
3. Select and fill the following options:
- File - find a local CSV file with the desired dataset
- CSV delimiter - defaults to "," but you can choose any other character
- File encoding - defaults to UTF-8
- Overwrite existing parameter values - if checked, this option will delete any values from existing parameters
- Create new parameters - If checked, it will create non-existing parameters automatically, based on the CSV column name. Otherwise, it will just append/update the values on the existing parameters.
4. Press "Import" to import the external dataset.
| Info | ||
|---|---|---|
| ||
You can also import combinational parameters. Any parameter with an asterisk suffix will be considered a combinational parameter. E.g., Quality*. In this case, the "Create new parameters" option must be enabled. |
Exporting a dataset to a CSV file
To export a dataset to a CSV file:
1. Open the Dataset dialog.
2. Click the options button next to the import button. Clicking this button will reveal a menu with options for Export to CSV and Delete.
3. Click the Export to CSV. A file should be downloaded.
| Info | ||
|---|---|---|
| ||
The delimiter used to generate the CSV file is a comma. The combinatorial parameters will be exported with an asterisk as a suffix |
Importing a dataset from a CSV file
Besides defining a dataset by creating parameters and setting their values directly using the Xray UI, it is also possible to import an existing dataset from a CSV file.
In order to import a CSV file:
1. Open the dataset dialog.
2. Choose the "Import from CSV" option available on the Import dropdown button (located on the empty state view or on the dataset main button toolbar). A dialog will appear.
3. Select and fill the following options:
- File - find a local CSV file with the desired dataset
- CSV delimiter - defaults to "," but you can choose any other character
- File encoding - defaults to UTF-8
- Overwrite existing parameter values - if checked, this option will delete any values from existing parameters
- Create new parameters - If checked, it will create non-existing parameters automatically, based on the CSV column name. Otherwise, it will just append/update the values on the existing parameters.
4. Press "Import" to import the external dataset.
| Info | ||
|---|---|---|
| ||
You can also import combinational parameters. Any parameter with an asterisk suffix will be considered a combinational parameter. E.g., Quality*. In this case, the "Create new parameters" option must be enabled. |
Overriding a dataset in a test plan or test execution
...