Page History

A common question from Test Case Designer clients in the early adoption stages is:

“Before, we had a clear testing focus for each scenario, so we knew exactly what would fail the test. Now, with all the inputs mixed together, how can we identify the cause of a defect?”

The purpose of this article is This article aims to demonstrate a defect analysis approach when using TCD tests. We’ll use a simplistic customer preferences application with 5 “Checked”/ ”Not Checked” indicators, looking like this:

...

So, when test #3 fails, it tells us there is likely something wrong with Indicator 3.

If instead, we use Test Case Desgner Designer to generate the a set of 2-way tests for this application, ; we might get test scenarios like these:

Now, if If scenario #4 fails, it’s not just Indicator 3 that is checked, but also Indicator 1.

...

First, let’s define the “strength” of the defect. We’ll call an issue a “one-way defect” if it is caused by just a single parameter value, irrespective of everything else happening in the test case. A “two-way defect” means that a specific combination of 2 parameter values happening together triggers the error (e.g., Checked for Indicator 1 AND Checked for Indicator 4), and so on for “three-way defects” and beyond.

...

If this is truly a 1-way defect of “Indicator 3 = Checked”, then the rest of the values in the row won’t matter. The same defect would be caught by each of the following highlighted TCD scenarios:

The debugging process, in this case, would consist of the following steps (assuming there is no error message specifying exactly what went wrong):

1. Execute all the scenarios and notice that scenarios #1, #4, and #6 were the only ones that failed.
2. Analyze what is the consistent element in each of these scenarios is:

• Indicator 1 has both values – not a culprit
• Indicator 2 has both values – not a culprit
• Indicator 3 has only “Checked” value – culprit
• Indicator 4 has both values – not a culprit
• Indicator 5 has both values – not a culprit

...

So it’s important to use a combinatorial testing methodology to improve the testing coverage and then adjust the defect analysis process accordingly.

...