“Before vs After” TCD Comparison Guide. The process description assumes intermediate knowledge of Test Case Designer features.

One of the most common concerns we hear from clients is:

...

That may be true, but we don’t have to guess. In this document, we describe the process to evaluate the existing tests, directly compare them with the tests generated by TCD, and make data-driven conclusions about what is best for the software testing efficiency in your organization. 

The example below uses a simple banking application, but this process can also be followed with any existing set of tests as long as it is converted to the parameterized data table. The order of creating the optimized Test Case Designer model or the one for analyzing the existing suite technically doesn’t matter (this article goes through the manual side first). It is crucial though that the models are exactly the same when it comes to Parameters/ Values and value expansions/ Constraints.

Table of Contents

Prerequisites

You will need to reorganize the existing tests in the TCD import format. It involves Parameters as columns and Values as rows:

...

• Direct duplicates (inconsistent formatting; spelling errors);
• “Hidden”/Contextual duplicates (meaningful typos; same instructions written by different people with varied styles);
• Tests specifying some values, leaving others as default (when several scenario combinations could be tested in the single execution run).

...

To generate the most precise comparison, the actual values from execution logs need to be placed in all the blanks in requirements (i.e. red font in the picture above). If that is impossible, the default value for each parameter is assumed to be used.

For this example, we use 8 artificial existing tests which didn’t specify all the values in their documentation.

Process

Next, we proceed with generating the comparison. First, we create the model with the goal of putting the reformatted dataset above onto the Forced Interactions tab. There are 2 options:  

...

Next, click “Scenarios” in the left navigation pane.

...

This is how your existing test suite looks when “generated” by TCD. However, we see that the algorithm believes you need 19 test cases (not 8 we imported in this example) to thoroughly explore the potential system weaknesses. Why?

...

Then the answer to the central question of this guide is on the Analysis screen.

Comparison & Conclusions

Remember the dangers of manual selection without the systematic approach? The “good enough” existing suite only covers 48% of 2-way interactions in the system, leaving a significant number of potentially-defect-causing gaps in coverage. 

...

What if you let Test Case Designer select all the non-specified values for the 8 business rules that you had? As you go to the Analysis tab in that copied model, this is what you should notice:

...

Test Case Designer is able to scientifically detect the optimal way to select values for each test scenario and generate 26% more interaction coverage with the same number of tests. Consequently, you hit the diminishing returns on coverage a lot sooner and your total suite size is smaller (17 in this case).

...