Page History
“Before vs After” TCD Comparison Guide. The process description assumes intermediate knowledge of Test Case Designer features.
One of the most common concerns we hear from clients is:
...
That may be true, but we don’t have to guess. In this document, we describe the process to evaluate the existing tests, directly compare them with the tests generated by DesignWiseTCD, and make data-driven conclusions about what is best for the software testing efficiency in your organization.
The example below uses a simple banking application, but this process can also be followed with any existing set of tests as long as it is converted to the parameterized data table. The order of creating the optimized DesignWise Test Case Designer model or the one for analyzing the existing suite technically doesn’t matter (this article goes through the manual side first). It is crucial though that the models are exactly the same when it comes to Parameters/ Values and value expansions/ Constraints.The process description assumes intermediate knowledge of DesignWise features.
Prerequisites
You will need to reorganize the existing tests in the DesignWise TCD import format. It involves Parameters as columns and Values as rows:
...
Note: there is a difference between “select these 3 values and everything else should be default for this rule” and “select these 3 values and everything else should can be anything because it doesn’t matter for this rule”. The second interpretation is much more common in our experience.
...
Next, we proceed with generating the comparison. First, we create the plan with model with the goal of putting the reformatted dataset above onto the Forced Interactions tab. There are 2 options:
- Manually create model with all the parameters & values then input each of the existing tests inside the tool on the Forced Interactions tab.
- Create an empty plan model with 2 dummy parameters, navigate to Forced Interactions, and use the cloud icon on the left for the Import dialog. It contains the template you should copy-paste the reformatted existing tests to.
Note: when working with large existing suites, making the updates in Excel and importing the file into DesignWise Test Case Designer is generally faster but let us know if you run into any issues.
...
This is how your existing test suite looks when “generated” by DesignWiseTCD. However However, we see that the algorithm believes you need 19 test cases (not 8 we imported in this example) to thoroughly explore the potential system weaknesses. Why?
...
Then the answer to the central question of this guide is on the Analysis screen.
Comparison & Conclusions
Remember the dangers of manual selection without the systematic approach? The “good enough” existing suite only covers 48% of 2-way interactions in the system, leaving a significant number of potentially-defect-causing gaps in coverage.
...
Thus, this portion of the comparison tells us that the existing thoroughness is not sufficient, and 4 more DesignWiseTCD-generated tests would be needed to get to 81% 2-way interactions, which is a safe benchmark proven by research studies. You can clearly see which pairs are still missing and make concrete execution decisions based on the business risks & constraints (i.e. execute all 19 tests to reach 100%).
However, that is not the key conclusion. These 2 images evaluate the concept of building DesignWise TCD tests on top of the existing ones to just close the coverage gaps. This approach ignores the potential benefits of completely remodeling the application inside DesignWiseTest Case Designer. Let’s prove the benefits of this alternate approach by looking at plan the model we copied (with removed forced interactions).
What if you let DesignWise Test Case Designer select all the non-specified values for the 8 business rules that you had? As you go to the Analysis tab in that copied model, this is what you should notice:
Note: we recommend opening the plans models in 2 different browser tabs so that you can easily go back and forth.
DesignWise Test Case Designer is able to scientifically detect the optimal way to select values for each test scenario and generate 26% more interaction coverage with the same number of tests. Consequently, you hit the diminishing returns on coverage a lot sooner and your total suite size is smaller (17 in this case).
This is the process to prove the objectively-superior nature of the DesignWiseTCD-selected tests. Did your results come out drastically different than the above ones? Please feel free to reach out to us and share your experience or ask for advice on putting together this comparison yourself.
...