When testers struggle to use TCD, identifying what factors and ideas should be included in the “Parameters” screen is the most common reason. To help address this challenge, this article describes several categories of variables that are often useful to include in software tests.
At the risk of stating the obvious, you’ll need to use your judgment here; not all of these categories will be necessary (or even useful!) to include in all your tests.
In this article, you'll learn how to identify & evaluate Parameter candidates in your systems under test and how to include them in your Test Case Designer models optimally.
An overview of variation ideas
Variables for software tests can be described in these general categories. It is important to remember that when you’re entering variables into the “Parameters” screen, you should include test inputs only. You should not include outcomes or expected results (as a general rule, we will talk about exceptions in the “How to handle Expected Results” article).
Environmental Variations
Environmental variations relate to different potential combinations of hardware and software that people might use and the different locations of data that might be pulled from when a given transaction is executed. Imagine you’re creating some end-to-end tests for a flight reservation system, like https://www.expedia.com/Flights. As a web application, you would probably want to include multiple hardware and software configurations combinations.
Hardware configuration inputs could include:
- Manufacturer: Dell, Lenovo, Apple
- Processor: Intel, AMD
- Age: manufactured within last 2 years, manufactured 2 or more years ago
Software configuration inputs could include:
- Operating System: Windows 7, Windows 8, Windows 10, Mac OS
- Browser: IE10, IE11, Firefox, Chrome, Safari
- Java settings: JavaScript enabled, JavaScript disabled
- Cookies: cookies enabled, cookies disabled
Channel inputs could include:
- Channel: typical web transaction, typical web transaction, phone call center (Note: double-weighting ‘typical web transaction’ will make it show up twice as often as ‘phone call center’)
Additional Environmental Variations involve file types and storage locations.
- Flight schedule information for flights is stored in: Database Huey, Database Dooey, and Database Louie.
- Frequent Flier Mileage data stored in United database, (Frontier Airlines database; not yet transferred into post-merger database), N/A
User Variations
User Variations relate to how different people might navigate the System Under Test based on their particular habits as computer users. They can also include, as in the case of an “Admin User” with special rights, the particular features of the system that they have available to them based on the kind of user they are.
For example, User Variations that may be considered in end-to-end tests of https://www.expedia.com/Flights might include:
- User type: normal user (without prior account), normal user (with established account), call center agent
- User persona: power user, clueless newbie user, the typical user
- Account characteristics: Gold Frequent Flier, Silver Frequent Flier, Bronze Frequent Flier, not a Frequent Flier
Other related ideas:
- The user navigates primarily using: keyboard (hot-keys wherever possible), keyboard (no hot-keys), mouse
- The user rapidly clicks multiple times on submit buttons and next buttons: no, yes, no (Note: double-weighting ‘no’ will make it appear twice as often as ‘yes’)
- The user enters information from the top of screen to bottom of the screen: always, usually, never
- The user enters all required information the first time they submit information: always, usually, never
Usage Variations
Usage Variations relate to what different features of a system people might use, as well as how they might use those features and the different types of data that might be used in different scenarios.
Actions that could vary from test to test in our https://www.expedia.com/Flights example could include:
- One Way or Round Trip: One Way, Round Trip
- Destination: South America, Europe, Asia, Africa, Australia, North America
- Class of Travel: Economy, Business, First
- Special Meal Requested: None, None, Vegetarian, Kosher (Note: double-weighting ‘None’ will make it show up twice as often as ‘Vegetarian’ or ‘Kosher’)
- Saturday Night Stay-over: Yes, No
Data that might vary will often be inextricably interrelated to usage examples (directly above). Each of the user selections above, for example, would create data that the system should store. That’s one of the reasons we categorize “Actions” and “Data” together as “Usage Variations.” Even so, “Data” can be useful as a trigger for additional testing ideas such as quantities, data ranges, and specific data formats. Here are some examples of test inputs that could be included in this example:
- Date format: mm/dd/yy, dd/mm/yy, mm/dd/yy, dd/mm/yyyy
- Number of passengers: 1, 2-4, more than 4
- Special characters included in customer name?: no, yes, no (Note: recall double-weighting)
Tips
“Hendrickson Variables”
“Hendrickson Variables” offer a great checklist to tick through as you think about possible variables to add.
Elisabeth Hendrickson published “Explore It!” in 2013. In our opinion, it is one of the best software testing books ever written. We highly recommend it to be new and experienced software testers alike without reservations. Chapter 4, “Finding Interesting Things to Vary,” is particularly outstanding. The chapter does a superb job of articulating what variables you can change from test to test and why it is such a good idea to vary them. These “Hendrickson Variables” come from Chapter 4 of “Explore It!”
Hendrickson lays-out her list of Interesting Things to Vary as follows:
- Countable Things
- Relative Positioning
- Files and Storage
- Formats
- Size
- Geographic Locations
- Depth
- Timing, Frequency, and Duration
- Input and Navigation
"Rule of thumb" based on the likelihood of relevance and effort
A “rule of thumb” to guide you as you decide what things should be included and excluded in the Parameters screen:
Deciding whether to include a specific test idea often comes down to a judgment call by the test designer using TCD.
INCLUDE – If you have reason to believe that including a test input is likely to matter and it would not take much extra time to vary that idea in each of the tests in your test set, then include it. An example: if you’re testing a transactional web application and prior releases have had quite a few defects associated with the IE10 browser, include “Browser Type” as a Parameter and “IE10” as one of your Values.
EXCLUDE – Conversely, if you suspect that a particular test idea will not make any difference. That variation would take considerable time to include in each of your tests; do not enter those into the “Parameters” screen of TCD. An example: as pointed out by James Bach, defects have been triggered by blocked cooling vents above servers on extremely rare occasions. This caused a server to overheat. Even so, it would rarely be sensible to invest the time necessary to test out multiple test scenarios with overheated servers on a given set of tests.
Including test ideas using the Value Expansion feature allows quick testing without wasting time (over-testing).
Example
In such situations, you might instead consider using “Type of Transaction” as a Value and then adding “Domestic, International, Domestic, Domestic, Domestic, Domestic, Domestic” as sub-values when you create your Value Expansion. Using this test design technique would result in at least one “International” transaction (for coverage of this potentially important scenario). Still, you would avoid spending too much time “over-testing” the idea.
When you’re first getting started, feel free to experiment!
Your ability to make judgment calls will increase as you gain experience. When you’re experimenting, don’t be afraid to add more test ideas into the scope of your test sets than you’re used to. You will notice that when you design tests with TCD, you can include more test ideas in the scope of your tests than you’re used to including. That’s because:
- You will often find surprising defects when you add variation to your tests, yet adding the variation to your tests often hardly takes any time at all.
- Adding a Parameter with a few Values takes only a few seconds on the “Parameters” screen of TCD.
- Finally, unlike when you select and document tests by hand, generating the tests with the newly-added test ideas won’t add noticeably more test documentation time; the tests generated by TCD will all automatically include that test idea.
- Provided you keep the number of Values per Parameter small, you usually won’t increase the number of tests that TCD will generate.