...

Adopting a more scientific strategy, as confirmed in multiple Test Case Designer case studies, helps eliminate large amounts of inefficiency – reducing the number of tests and making the test suites more easily maintainable. This approach

Approach Summary

Such combinatorial, model-based testing can be applied beyond BAU regression optimization to scenarios like new feature releases or applications going through a redesign.

The “best practice” process to follow is shown on these flow diagrams:

Image RemovedImage Added

It can be aggregated into the “cheat sheet” with four key stages.

Stage 1: Begin test design efforts focusing on potential variation in the system under test

Instead of focusing only on details specified in requirements documentation, it is crucial to take a step back and evaluate what matters for the system as a whole. It is necessary to analyze which steps a user could take in an application and which choices they could have at each step.

Then it is important to think about external elements that could affect user behavior (dependencies with other applications, environments, etc.).

Finally, testers should organize that information in the appropriate form, like parameter/value tables, in order to review this information with stakeholders. Test Case Designer mind maps (first image on the top diagram above) can also be useful to facilitate such discussions.

Once all the key input parts of the model have been agreed upon, the team should proceed to the analysis of the constraints (i.e. how inputs can and can’t interact with each other) and requirements (both formal and informal).

Stage 2: Create systematically-varied, data-driven test scenarios

Next, test designers should generate efficient and thorough test scenarios based on the inputs defined in the model.

It is beneficial to use Combinatorial Test Design methodologies at this step as they apply selection algorithms to identify the scenario components that guarantee maximum variation in the minimum necessary number of tests.

This process can be difficult and time consuming to accomplish manually but a proprietary combinatorial algorithm helps TCD simplify and accelerate this part of the process.

...

Further, it’s crucial to understand the level of interaction coverage of the test suite as on average 84% of defects in production are caused by 1 or 2 system elements acting together in a certain way.

Evaluating the test suite at the model level leads to risk reduction through a clear understanding of what exactly is covered in each set of tests (for example, by leveraging the coverage graph & matrix in Test Case Designer).

Stage 3: Create consistent and accurate documentation

Increasing the consistency and reducing the ambiguity of both script steps and expected results improves execution time & effort and makes automation more straightforward. When considering the expected results, it is important to precisely identify which combination of inputs trigger each desired outcome.

TCD allows you to create a single script for the model that is automatically applied to each of your test scenarios. This allows for the best practice of creating data-driven test scripts and ensures the script creation effort does not increase linearly with each test scenario. The tool also allows users to incorporate the appropriate logic for expected result generation.

Stage 4: Simplify test case maintenance over time

One of the key advantages of data-driven testing is the ability to make a single update to the model’s inputs and have that update automatically applied to all of the test scenarios. This saves significant time and effort maintaining a test suite over time and helps ensure consistency and accuracy of all test cases. TCD also allows you to export the updated artifacts in a variety of formats for use with test case management and test automation tools.

...