Page History
How to identify the defect reason after you move to combinatorial testing? When teams start exploring DesignWise, they are often unsure what will happen to the plans after the initial release – will they have long-term value, where is traceability with execution, how to conduct maintenance, etc.? In this article, we have consolidated the “cheat sheet” from the information available in our feature-focused documents and provided additional insights gained from client projects.
Challenge Summary
A common question from DesignWise Test Case Designer clients in early adoption stages is:
...
The purpose of this article is to demonstrate a defect analysis approach when using DesignWise TCD tests. We’ll use a simplistic customer preferences application with 5 “Checked”/ ”Not Checked” indicators, looking like this:
...
So, when test #3 fails, it tells us there is likely something wrong with Indicator 3.
If instead we use DesignWise Test Case Desgner to generate the a set of 2-way tests for this application, we might get test scenarios like these:
...
Now, if scenario #4 fails, it’s not just Indicator 3 that is checked, but also Indicator 1. So, how do we pinpoint the reason for the failure?
An Analysis Approach
The key to our analysis is to look at the whole test suite and identify the dependency pattern between the consistent elements and the execution outcome.
...
If this is truly a 1-way defect of “Indicator 3 = Checked”, then the rest of the values in the row won’t matter. The same defect would be caught by each of the following highlighted DesignWise scenariosTCD scenarios:
The debugging process in this case would consist of the following steps (assuming there is no error message specifying exactly what went wrong):
...
As in this case, the holistic view of the DesignWise scenarios Test Case Designer scenarios usually provides you with enough data to pinpoint an n-way issue.
...
So it’s important to use a combinatorial testing methodology to improve the testing coverage and then adjust the defect analysis process accordingly.
Achieving both isolated and combinatorial testing
There may be situations where targeted testing is absolutely necessary. DesignWise Test Case Designer allows you to do this by forcing specific parameter values to be used. For example, we can force only 1 indicator at a time to be checked in each of the first 5 scenarios:
Then the rest of the DesignWiseTCD-generated scenarios will provide all of the remaining pairwise interaction coverage.
...