Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Nobody wants to perform testing endlessly, without any sort of criteria. Random testing doesn't ensure we cover combinations that matter with a very limited set of tests.

Empyrical data about fault detection

Several studies indicate that the vast majority of defects (67%-84%) related to input values are due to either to a problem in a parameter value (single-value fault) or in a combination of two parameter values (2-way interaction fault).

Single-value faults are mostly probable to typical mistakes, such as the off-by-one bug (e.g., imagine using a loop and using the "<" operator instead of "<="). The interaction of 2 parameters may be to bugs around implementing cascade conditional logic statements (e.g. using if or similarinvolving those parameters/variables.

Bugs related to the interaction of more parameters decrease with the number of parameters; in other words, finding these rare bugs will require much more tests to be performed, leading to more time/costs. However, those rare t-way interaction faults can also be critical.

Pairwise (2-way) and t-way Testing

...

Xray has built-in support for parameterized tests and datasets, supporting user-defined datasets and automatic generation of combinations for the identified parameters.

Test Case Designer (TCD) provides a more comprehensive modeling tool, where it's possible to define parameters, apply constraints, enforce specific scenarios, generate optimized datasets/scenarios using 2-way or n-way to ensure that certain parameters are tested together, up to a certain level. With Test Case Designer it's possible to have a limited and manageabled set of test scenarios to perform and make sure that most combination of parameters are met with the initial scenarios, so that most risk is addressed upfront.

TCD doesn't replace Xray built-in capabilities for parameterized tests and datasets; it's a more evolved approach. Both can be used in a given project.

...