Page History
...
This leads to a potential high-number of scenarios to be tested. For the previous example, and considering a model where parameters have just a limited number of possible values, that would still lead to 3*3*3*2*3=162 scenarios.
| Info |
|---|
System produce different results not only due to changes on the input parameters but also due to using the system on different contexts (e.g., configurations, operating system, timezones, cloud provider). |
In general, we could think that if we aim to test systems like this in-depth, we would need to test the system with all the possible combinations of values of these parameters.
...
| Info | ||
|---|---|---|
| ||
In Test Case Designer we can . In the following example, we considered an interaction that we need to test due to an hypothetic legislation where some warning must be shown to users who are departing from USA, using the First class, and have more than 1 children. That scenario will be added on the generated ones.
|
...
Challenges
Pairwise or t-wise testing even though useful, it's not a silver bullet.
Some challenges or limitations to be aware of include:
- test oracle: this technique doesn't address finding the proper test oracle for the generated scenarios. How do we know the scenario is behaving as expected? How do we know that a given scenario has issues or not?
- modeling: depicting a "good" model requires the intervention of testers. Testers with the help of other team members are the ones able to figure out representative and important scenarios to model , their parameters, the values for those parameters, constraints, etc.
Using pairwise and t-wise for scripted testing and exploratory testing
Whenever generating an optimized dataset (i.e., multiple "rows" of values for the parameters) this will be typically used to data-drive a scripted test case (e.g., a "manual" test composed of steps, or an automated test script).
In that case, testers would specify the steps to follow and include references to the parameters on those steps. To perform testing, the test is iterated multiple times, as many as of the generated dataset rows (i.e. combination of parameter/values). In each iteration the parameters are replaced by the corresponding values on the dataset row.
Generating these combinations is useful not only for this testing approach though.
Pairwise and t-wise testing don't tell us how to actually perform testing; it just generates the combination of parameters. Therefore, we can use this technique also if choose to adopt a more exploratory testing approach, for example for certain configurations of hardware/software. include:
Xray datasets and Xray Test Case Designer
...