Introduction
Test parameterization is a powerful practice that allows the same test to be executed multiple times with different parameters. Parameters are similar to input values (variables) that can change with each execution.
Without the ability to define parameterized Tests, all values must be hard-coded into the specification, making the Test static and difficult to modify. Static Tests lead to redundancy since you need to define and clone the same Test for different combinations of values to cover various scenarios or variations of the same Test.
By extracting the variability of the Test specification into a table of test inputs and verifiable outputs, you effectively data-driven testing. Ideally, these Tests are automated. However, this might not always be possible or viable, and manual Tests can also benefit from this methodology.
Parameterized Tests vs Test Environments
Although Test environments can be considered a particular case of parameterized Tests (since environments can be seen as parameters of Test cases), datasets might not be the best choice for environmental variables that are not embedded in the Test specification (e.g., manual Test Steps).
It is not good practice to use datasets with environment variables such as browsers, operating systems (OSs), databases, or devices. This is because:
Test engineers often configure a specific setup (or environment) to execute a set of Tests. In this case, they might want to execute the Tests oriented to the environment rather than the Test case. This means executing all the Test cases first for one environment, then for another, and so on. Having all these environments within the same Test run is not ideal as users would have to jump from one Test run to another without finishing the execution.
Test environments are usually independent variables or dimensions. For instance, consider the following Test environment: browser, database, and OS. If a Test case fails only in a specific browser, once the bug is fixed, you might not want to re-execute the same Test for all databases and OSs, as you are confident that the change did not affect these variables. With datasets, if one iteration fails, you need to re-execute all the iterations again (assuming you use a different Test run, of course).
Xray does not provide reports based on parameters.
In conclusion, if your parameters are environment variables that do not need to be included in the test specification, don't use datasets. Instead, use test environments.
We plan to improve how test environments are managed in Xray, making it possible to specify or generate combinations of environments for different variables.
Working with Parameterized Tests
Example
Suppose you need to validate the login on a website with a set of valid and invalid usernames and passwords:
| Username | Password | Valid |
|---|---|---|
| admin | 123123 | valid |
| john.doe | ###### | invalid |
| jane.doe | jane123 | valid |
The following Test cases need to be executed:
| # | Action | Data | Expected Result |
|---|---|---|---|
| 1 | Open the website | N/A | The main page is displayed and the user can enter login credentials |
| 2 | Enter the following login and password, and click the Login button | Login: admin Password: 123123 | The login is valid |
| # | Action | Data | Expected Result |
|---|---|---|---|
| 1 | Open the website | N/A | The main page is displayed and the user can enter login credentials |
| 2 | Enter the following login and password, and click the Login button | Login: john.doe Password: ###### | The login is invalid |
| # | Action | Data | Expected Result |
|---|---|---|---|
| 1 | Open the website | N/A | The main page is displayed and the user can enter login credentials |
| 2 | Enter the following login and password, and click the Login button | Login: jane.doe Password: jane123 | The login is valid |
Instead of creating separate Tests, Test designers can instead create a single Test with the following parameters: Username, Password, and Valid.
| # | Action | Data | Expected Result |
|---|---|---|---|
| 1 | Open the website | N/A | The main page is displayed and the user can enter login credentials |
| 2 | Enter the following login and password, and click the Login button | Login: ${Username} Password: ${Password} | The login is ${Valid}. |
Parameterized Tests in Xray
Parameterized Tests in Xray (Figure 1) are defined just like any other Test with the addition of some parameter names within the specification.
Figure 1 - Parameterized tests
Parameters are embedded within the Test specifications using the following notation: ${PARAMETER_NAME}.
Parameter names are case-sensitive.
This notation (Figure 1 - 1) is used to reference parameters within the Test Steps. You can reference parameters in the Action, Data, Expected Result fields, and any text-based custom fields in the Test Steps.
Parameters are defined within datasets. However, it is possible to reference a parameter that is not yet defined (meaning it does not have a corresponding name within the dataset). In this case, the parameter will be highlighted in red (Figure 1 - 1).
Currently, the Test parameterization feature is only available for Manual Tests. Xray will support defining parameters and use datasets on BDD and Generic Test types as well.
Defining a Parameterized Test
To reference parameters within Test steps:
Create or edit a Test Step using either the inline view within the Test Issue or the Steps modal.
When specifying a Test Step, to reference a parameter you have two options:
- Start typing
${(Figure 2 - 1). If there is a default dataset defined on the Test, you will see a list of the available parameters. Choose the desired parameter using the cursor keys or mouse. The parameter will be inserted into the text. - Use the toolbar button
${(Figure 2 - 2). After pressing this button, and if there is a default dataset defined on the test, you will see a list of the available parameters. Choose the desired parameter using the cursor keys or mouse. The parameter will be placed at the cursor position.
Figure 2 - Defining parameters
Once you're finished, click the Save button (Figure 2 - 3).
Datasets
Parameters and their values are defined within a dataset. A dataset is a collection of data represented in a tabular view, where every column of the table represents a particular variable (or parameter), and each row corresponds to a given record (or iteration) of the dataset.
The number of rows in the dataset determines the number of iterations to execute. If the dataset contains a single row, there will be a single execution parameterized with the values defined in the dataset row.
Datasets can be defined in different entities and scopes. A dataset can be defined, edited, or simply viewed using the Dataset button (Figure 2 - 4) located in each Xray entity or scope.
Parameter Types
Parameters can have the following types:
- Text - the parameter value will be set using an open text field.
- List - the parameter value can be selected from a predefined list of options. They can be created using either ad hoc or predefined lists.
Ad Hoc Lists
Ad hoc lists are defined locally for each parameter, while predefined lists are created by administrators at different levels:
- Global - managed by Jira administrators. Global lists can not be used directly. These must be included in the project by the project administrator before they can be accessed within a dataset of that project.
- Project - managed by project administrators.
Predefined Lists
Predefined lists is useful if the list parameters are commonly used in multiple datasets. If multiple projects make use of the same list, you can also create a global list so that it can be used by different projects. This way you have a central place to manage common parameter lists. Examples of predefined lists include:
- Profiles.
- Users.
- Roles.
- Colors.
- Credit card types.
- Addresses.
- Etc.
When creating a new list parameter using a predefined list, you can choose a list that is available within the current project. The current project is determined by the parent issue where the dataset is defined.
Combinatorial Parameters
Combinatorial parameters are special parameters that will be combined with the remaining parameters (combinatorial or seeding parameters) to generate all possible combinations automatically. This prevents users from typing all the combinations when creating a dataset.
Seeding parameters are those parameters that describe fixed Test cases. The seeding parameters will not be combined with each other; they will only be combined with combinatorial parameters.
Example
This is a test to see if you can add books to a shopping cart in your online bookstore. The parameters are: Item, Price, Rating, In Stock, Condition, and Format. There are certain books to be tested (three in this case). However, we will test all combinations of these books with the following parameters: Gift and Quantity.
In this case, these will be the seeding parameters (Figure 3 - 1):
- Item.
- Price.
- Rating.
- In Stock,
- Condition.
- Format.
Since we want to test these items with all the combinations of Gift and Quantity parameters, we can create these as combinatorial parameters (Figure 3 - 2):
- Gift*
- Quantity*
Combinatorial parameters are denoted with an asterisk (*) suffix.
Figure 3 - Parameters
Xray will generate all possible combinations upon execution automatically (Figure 4).
Figure 4 - Combinations
Dataset Scopes
A dataset can be defined in the following entities/scopes:
- Test (default dataset). If there is the need to override or change this dataset, you can do this at the Test planning or Test execution phases.
- Test Plan - Test.
- Test Execution - Test (Test Run).
The closest dataset to the Test run will be the one used to generate the iterations, effectively overriding any dataset defined at higher levels:
Test Execution - Test (Test Run) > Test Plan - Test > Test (default)
To define/see the dataset scopes, click the Dataset icon (Figure 5 - 1) and/or the Actions button (Figure 5 - 2), and then select the Dataset option (Figure 5 - 3).
Figure 5 - Scopes
Dataset Limits
- Maximum number of iterations per dataset: 1000.
- Maximum Number of parameters per dataset: 20.
- A dataset can not contain duplicate rows.
Creating a Dataset
To create or edit the default dataset (within a Test):
Click the Dataset button (Figure 6 - 1). A modal will open (Figure 7).
Figure 6 - Dataset
Click the Create parameter button (Figure 7 - 1). A modal will open for you to specify parameter attributes (Figure 8).
Figure 7 - Parameter
Here (Figure 8), you can:
- Specify the parameter's name (Figure 8 - 1; mandatory field). Parameter names must start with a letter or underscore and can only contain letters, numbers, a space between words, "_", "-" and a maximum of 64 characters.
- Check the Combinatorial checkbox if you are creating a combinatorial parameter (Figure 8 - 2).
- Choose the parameter type: Text or List (Figure 8 - 3). If the parameter type is a List, you can:
- Create an Ad hoc list just for this parameter (Figure 8 - 4). You need to specify the values for the list (Figure 8 - 5; mandatory field).
- Use a Project predefined list (Figure 8 - 4).
- Once you're finished, click Create (Figure 8 - 6). You will be redirected to the dataset modal (Figure 9).
Figure 8 - Parameter
Once you're finished, click Save (Figure 9 - 2). The parameter will be placed in the dataset.
Figure 9 - Modal
More Operations
Adding Combinatorial Parameter Values
Once you have at least one parameter, you can start filling in their values and adding new iterations.
A placeholder is provided within each combinatorial parameter. To add new values to combinatorial parameters:
For text parameters, type the value and click the check icon (Figure 10 - 1).
Figure 10 - Text
For list parameters, select an option and click the check icon (Figure 11 - 1).
Figure 11 - List
Adding Rows (Filling the Parameter Values)
Once you have non-combinatorial parameters, an empty placeholder row will appear so that the parameters can be populated for the default iteration (Figure 12).
Editing parameter values is as simple as editing their corresponding cells. The values will be kept when the cell loses focus.
You can navigate between cells of the same row and also between rows using the keyboard: TAB (forward), SHIFT+TAB (backward).
To create new rows, you can click the New button (Figure 12 - 1), or navigate using the keyboard from the last row (a new row will be created automatically by navigating forward to the last cell of the last row).
Figure 12 - Parameters
Converting a Seeding Parameter into a Combinatorial Parameter
You can convert an existing (seeding) parameter into a combinatorial parameter. This will remove the parameter column from the seeding parameters table and group all remaining rows automatically. A new combinatorial parameter will be created by grouping all the values.
1 - Next to each column, there is an ellipsis button (Figure 13 - 1). Clicking this button will reveal a menu with options for Editing, Deleting, and Converting parameters (Figure 13 - 2).
2 - Click the Convert to combinatorial parameter (Figure 13 - 2) option.
Figure 13 - Menu
3 - A modal will open (Figure 14). Verify the parameters (Figure 14 - 1) and once you're finished, click Save (Figure 14 - 2).
Figure 14 - Parameters
Converting a Combinatorial Parameter into a Seeding Parameter
You can convert a combinatorial parameter back into a seeding parameter.
1 - Click the ellipsis button (Figure 15 - 1). Then, select Convert to non-combinatorial parameter (Figure 15 - 3).
2 - Once you're finished, click Save (Figure 15 - 4).
Figure 15 - Parameters
Generating all Combinations
You don't need to generate all the combinations for a dataset to execute all iterations. Xray will do this automatically for all of the combinatorial parameters. However, if you don't need all the combinations, you can generate all combinations and remove some iterations afterward.
1 - Click the ellipsis button (Figure 15 - 1). Then, select Generate all combinations (Figure 15 - 2).
2 - A modal will open for you to proceed. Click Confirm (Figure 16 - 1). Confirming the changes will apply the Cartesian product between the combinational parameter values and the seeding parameter rows.
Figure 16 - Modal
3 - Once you're finished, click Save (Figure 16 - 2).
Moving Parameters Left/Right
You can reorder any parameters in your dataset, by clicking the ellipsis button (Figure 17 - 1) and selecting the Move left/right option. This will swap the parameters' positions.
Figure 17 - Parameters
After making any changes to the dataset, click the Save button (Figure 16 - 2) to keep the dataset information in the database updated.
Importing a Dataset from a CSV File
Besides defining a dataset by creating parameters and setting their values directly using the Xray UI, it is also possible to import an existing dataset from a CSV file.
On the Test Issue, click the Dataset icon (Figure 18 - 1). A modal will open (Figure 19).
Figure 18 - Dataset
Click the Import button (Figure 19 - 1) and select the CSV file option (Figure 19 - 2). A modal will open (Figure 20).
Figure 19 - Import
In the Import from CSV modal, select and fill in the fields (Figure 20):
- Choose file - click this button to find a local CSV file with the desired dataset (Figure 20 - 1).
- CSV Delimiter - defaults to "," but you can choose any other character (Figure 20 - 2).
- File Encoding - defaults to UTF-8 (Figure 20 - 2).
- Overwrite existing parameter values - if enabled, this option will delete any values from existing parameters (Figure 20 - 3).
- Create new parameters - if enabled, it will create non-existing parameters automatically, based on the CSV column name. Otherwise, it will just append/update the values on the existing parameters (Figure 20 - 3).
Figure 20 - CSV
Once you're finished, click the Import button (Figure 20 - 4).
You can also import combinational parameters. Any parameter with an asterisk suffix will be considered a combinational parameter. E.g., Quality*. In this case, the Create new parameters option (Figure 20 - 3) must be enabled.
Exporting a Dataset to a CSV File
On the Test Issue, click the Dataset icon (Figure 21 - 1). A Dataset modal will open (Figure 22).
Figure 21 - Dataset
In the Dataset modal (Figure 22), click the ellipsis button (Figure 22 - 1), and select the Export to CSV option (Figure 22 - 2).
Figure 22 - Export
The file download will start right away and you will get the file on your machine.
The delimiter used to generate the CSV file is a comma.
Deleting a Dataset
On the Test Issue, click the Dataset icon (Figure 21 - 1). The Dataset modal will open (Figure 22).
In the Dataset modal, click the ellipsis button (Figure 22 - 1) and select the Delete option (Figure 22 - 2). A modal will open for you to confirm the removal of the dataset (Figure 23). Click Confirm to proceed (Figure 23 - 1).
Figure 23 - Delete
Parameterized Preconditions
Precondition Issues can also be parameterized by including parameter names in the precondition specification (Figure 24).
The parameters will be unfolded on the Test execution screen, just like in Test cases. For this, the dataset must have the same parameters, matched by name.
Figure 24 - Precondition
Iterations Execution
All iterations for a given Test are executed within the context of the same Test run. Each iteration can be expanded by clicking the arrow icon (Figure 25 - 1), and the Test Steps executed individually (Figure 25 - 2). The Step parameters will be replaced by the corresponding iteration values. The Steps affect the iteration status which, in turn, affects the overall Test run status (Figure 25 - 3).
Figure 25 - Iterations
Overriding a Dataset in a Test Plan or Test Execution
Navigate to the desired Test plan or Test execution Issue (Figure 26).
A menu action is provided on each Test row (Figure 26 - 2). A column named Dataset (Figure 26 - 1) is also available by configuring the column layout for the Test Plan or Test Execution datatable. If there is a dataset already defined at this level, the Dataset button (Figure 26 - 1) will be displayed with a selected style.
Figure 26 - Execution
Click the Dataset button (Figure 26 - 1) or click the Actions button (Figure 26 - 2) and then select Dataset (Figure 26 - 3). The Dataset modal will open (Figure 27).
In the Dataset modal (Figure 27), if there is a dataset defined on a parent level, you can override the parent dataset and modify its values by clicking Override (Figure 27 - 1). Otherwise, you can start defining a new dataset at this level (Figure 27 - 2).
Figure 27 - Override
Once the dataset is defined (or overridden), click Save (Figure 27 - 3).