Page History
| Table of Contents |
|---|
Overview
Systems in general, including software and hardware, usually have some input parameters that will affect the output produced by the system. The number of parameters and the possible values, their ranges, vary and can be limited, huge but finite, or even infinite.
...
In this tutorial we'll learn about the testing challenges of these systems and how to overcome them efficiently.
Initial testing options
Test using some examples for the parameters
The first strategy that we may come with would be adopting data-driven testing.
...
| Info | ||
|---|---|---|
| ||
Xray has built-in support for datasets where testers can explicitly enumerate parameters and the combination of values to be tested.
Please see Parameterized Tests for more info. |
Test every parameter/value combination
Testing every possible combination of parameters is only viable if we have very few parameters with very few possible values for each one of them.
...
| Info | ||
|---|---|---|
| ||
Xray also supports combinatorial parameters, where the user defines the values for each parameter and Xray calculates all the possible combinations, turning that into the dataset to be used.
It's possible to remove some values of the combinations to be generated. For example, we can exclude the "First" Class. That would lead to less scenarios to test (e.g., 162 => 108) but could still not be enough if we aim to have a limited set of tests.
Please see Parameterized Tests for more info. |
Test using random combination of parameter values
Random testing is always an option that comes but it doesn't ensure we test combinations that matter unless we perform a very high number of tests, which would probabilisticly include a certain % of combinations or even all of them if we spend an infinite time randomly testing.
Nobody wants to perform testing endlessly, without any sort of criteria. Random testing doesn't ensure we cover combinations that matter with a very limited set of tests.
Empyrical data
Several studies indicate that the vast majority of defects (67%-84%) related to input values are due to either to a problem in a parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).
...
Bugs related to the interaction of more parameters decrease with the number of parameters; in other words, finding these rare bugs will require much more tests to be performed, leading to more time/costs.
Pairwise and n-wise Testing
Given the empyrical data, adopting pairwise testing to test all the ombinations of pairs of parameters (sometimes also called as "all pairs testing") is a technique that is not only feasible but also provides great results in terms of fault-detection.
Reducing the number of test scenarios
Imagining the previous example, instead of having XXX test scenarios to perform, we would need just XX.
...
| Info | ||
|---|---|---|
| ||
Let's say that we have 5 parameters. In this case, 5-way testing would generate all the possible combinations of these 5 parameters. Therefore, 5-way testing is precisely the same as saying that we're going to test all combination of parameter values. |
Optmizing further the test scenarios to be performed
The first level of optimization is further reducing the number of generated test scenario.
...
| Info | ||
|---|---|---|
| ||
In Test Case Designer we can . In the following example, we considered an interaction that we need to test due to an hypothetic legislation where some warning must be shown to users who are departing from USA, using the First class, and have more than 1 children. That scenario will be added on the generated ones.
|
Challenges
Pairwise or t-wise testing even though useful, it's not a silver bullet.
...
- test oracle: this technique doesn't address finding the proper test oracle for the generated scenarios. How do we know the scenario is behaving as expected? How do we know that a given scenario has issues or not?
- modeling: depicting a "good" model requires the intervention of testers. Testers with the help of other team members are the ones able to figure out representative and important scenarios to model , their parameters, the values for those parameters, constraints, etc.
Using pairwise and t-wise for scripted testing and exploratory testing
Whenever generating an optimized dataset (i.e., multiple "rows" of values for the parameters) this will be typically used to data-drive a scripted test case (e.g., a "manual" test composed of steps, or an automated test script).
...
Pairwise and t-wise testing don't tell us how to actually perform testing; it just generates the combination of parameters. Therefore, we can use this technique also if choose to adopt a more exploratory testing approach, for example for certain configurations of hardware/software.
Xray datasets and Xray Test Case Designer
Xray (available in all Xray versions) | Xray Test Case Designer (part of Xray Enterprise only) | |
|---|---|---|
| Parameters | ||
| define parameters | x | x |
| parameters: enumerate possible values | x | x |
| parameters: range of values | - | x |
custom dataset (i.e., enumeration of values for all parameters) | x | - |
| generation of all combinations of parameters/values | x | x |
| generation of a partial combination of parameters | x | - |
| smart generation of scenarios using pairwise (2-way testing) | - | x |
| smart generation of scenarios using n-way testing | - | x |
| constraints/rules on generation of scenarios | - | x |
| forced interactions | - | x |
| Creation of tests using generated data | ||
| authoring test cases (definition of steps) using the generated data | x | x |
| generation of test automation code skeleton for multiple testing frameworks, using the generated data | - | x |
| Reporting | ||
| tracking n-way coverage | - | x |
...