Introduction and accessing the scripts

Scripts allow you to define the steps and expected results for your Scenarios. They provide clear instructions for testers or automated frameworks, ensuring consistency in how scenarios are executed and making your tests more reliable and well-documented.

Since these scripts are data-driven and are based on a test model, they are more maintainable. Adding or removing parameter values, constraints, or forced interactions in your model updates the Scenarios data driving the Scripts, but there is often no need to adjust the script details.

You can quickly transform optimized test data as depicted in Figure 1 into customizable scripts (Figures 2 and 3).

Figure 1 - Scenarios Data

Figure 2 - Manual Scripts

Figure 3 - Automate Scripts

Let’s see how we can do that. To access this feature, open a model - e.g. click My Test Models (Figure 4 - 1) and then click any of them (Figure 4 - 2) - or create a new one.

Figure 4 - Open Test Model

A menu will appear on the left. There, click Scripts (Figure 5 - 1) and select one of the two options: Automate (Figure 5 - 2) or Manual (Figure 5 - 3).

Figure 5 - Scripts menu navigation

Automate

Automate (Figure 5 - 2) is a smart Gherkin feature file editor. It is “aware” of the parameters, values, and generated test cases of the current model as well as the Gherkin syntax, with highlighting and autocomplete features. Automate makes it easy to create a feature file with data-driven Scenario and/or Scenario Outline sections that are populated by the generated test cases of your model.

Note: we’ll try to make the distinction between the Scenarios screen - with its scriptless data table - and the Scenario/Scenario Outline options of Automate scripts as clear as possible.

Working on an Automate Script

Top ribbon

Figure 6 - Top ribbon functions in Automate

Work on your Gherkin feature in the editor panel on the left (Figure 6 - 1), and the generated Scenario sections or Examples data tables are shown in the preview panel on the right (Figure 6 - 2). Once you're done editing, click the Save button (Figure 6 - 3). This button will become active as soon as you make a change in the text field (Figure 6 - 1). You must generate test cases (Scenarios data table) first before the preview becomes active, then select the appropriate strength (Figure 6 - 4). 

You can click the Usage button (Figure 6 - 5) for a refresher on the Automate functionality.

Feature and Background

Each Gherkin script has a Feature as its first keyword followed by : and a little bit of context to describe the scope, usually in the form of a user story (which is demonstrated in the placeholder text). This section does not support parameterization of any kind.

A Background section is useful when a set of initial setup steps is repeated before each Scenario and Scenario Outline in the feature file. Starting a new line in Background and further sections brings up a suggestion dialog with standard Gherkin keywords. Keep in mind that restricting a parameter in the Background (with {[]}, as explained in the Conditional Logic section below) is valid syntax and is used to apply only the selected value to all Scenarios or Scenario Outlines for a given Feature.

Scenario or Scenario Outline

Next in the feature file is the enumeration of different testing scenarios. One way to describe a test case is with a Scenario block. It contains sequential steps, expressed as plain English Given/When/Then statements, that describe the necessary actions for executing the automated test script. If the model generates 28 rows on the Scenarios screen, then you could think that 28 Gherkin Scenario sections would need to be written out in order to achieve the combinatorial coverage.

However, instead of writing out all 28 scripts individually, you leverage Automate to create a single data-driven Scenario that includes the <Parameter Name> syntax. Then the tool can generate all 28 needed Scenario sections in the exported feature file. You can check the number of associated test cases in the Preview panel (Figure 7).

Figure 7 - Scenario block example in Automate

A Scenario Outline behaves similarly to a data-driven Scenario. When using a Scenario Outline, instead of having 28 Scenario sections (one for each test case), the export has one data-driven Scenario Outline with a Gherkin Examples data table attached to it that contains 28 rows (again, one for each test case). The Preview panel shows the first 10 rows of the Examples data table. The order of parameters in the Examples is dictated by the script, not by the Parameters screen of your model.

Figure 8 - Scenario Outline block example in Automate

Note: where possible, using Scenario Outline sections is preferable to generating many data-driven Scenario ones as Outline is a more compact expression of the testing ideas of the model.

The basic principles of your work with an Automate script are similar to those in any other Gherkin text editor. Click the grey background in the left panel to edit the script's text and write anything you want. You can phrase the actions however you’d like and utilize any combination of Gherkin keywords. With that said, (in most cases) you must connect Scripts to the Scenarios table via the parameterized syntax.

Simply typing < will populate a list of autocomplete suggestions related to all of the parameters in the model. At the end of the list, we have system variables:

• Test Number (sequential number within the Scenario script block),
• Test Case (constant number of the row in the complete table on the Scenarios screen)
• Expected Outcome (from the Forced Interactions screen, if applicable)

Wherever the <Parameter Name> appears in the test steps it will be replaced by the specified parameter's value for the corresponding test case. For example, writing the step:

      And they select <Color> for the car's color

means that in each generated Scenario script block, the <Color> portion of the step will be replaced with the value of the Color parameter for the corresponding test case:

      And they select Deep Blue Metallic for the car's color

      or

      And they select Red for the car’s color

      etc.

Note: it is common to put dynamic elements in quotes (e.g., And they select "<Color>" for the car's color) to make them more visually distinguishable.

As you insert parameters into the script, they will become greyed out in the dropdown you get from typing <. It’s a quick way to check that you haven’t accidentally omitted any important variables by the time the script is finished.

Automate - Conditional Logic

Oftentimes application flows may have more or fewer steps that need to be executed for specific test cases. For example, in Tesla's car configurator, if the user is configuring a new Tesla Model S, there is no option for the Interior Layout. 

Let’s say our test model for the Tesla car configurator contains a bi-directional bound pair constraint tying Model = Model S and Interior Layout = N/A together. If we don’t do anything else and just pass <Model> and <Interior Layout> to an Automate script, some test cases will read:

      And they select the N/A Interior Layout

While you could figure out what this means in manual testing or set up an extra code snippet in automated testing, it’s a bit suboptimal.

Automate provides a way to restrict a Scenario or Scenario Outline to test cases that contain a specific value using the {Parameter Name[Parameter Value]} syntax. Simply typing { will populate a list of autocomplete suggestions related to all of the parameter values in the model.

Figure 9 - Example of a conditional restriction in Automate

Including {Parameter Name[Parameter Value]} anywhere in the set of steps in a Scenario or Scenario Outline will restrict the test cases included in the generated Scenario blocks or Examples data table to only those that contain the specified parameter value.

So, for instance, you would create 1 Scenario block with {Model[Model X]} and with the Interior Layout step + 1 Scenario block with {Model[Model S]} and without the Interior Layout step.

As mentioned earlier, if all script blocks within an Automate tab follow the same conditional logic, you can restrict all of them in one action by using the {Parameter[Parameter Value]} syntax in the Background section

You can have more than 1 value per restriction, just separate them with commas. It is also possible to invert a restriction by adding an exclamation point just after the opening curly bracket, like so - {!Parameter[Value 1, Value 2]} This includes all the test cases in which "Parameter" is not "Value 1" or "Value 2".

IMPORTANT -  Please make sure that you cover all the generated rows from the Scenarios screen across the Value Restrictions, especially if you leverage multiple Automate tabs.

Automate - Multi-tab Approach

Each script tab in Automate corresponds to 1 .feature file (or 1 .csv file for transitions to certain test management tools like Xray). 

An Automate tab can contain as many data-driven Scenario and/or Scenario Outline sections as needed to automate all the testing ideas and flows contained in the test model. For the avoidance of doubt, you can mix Scenario and Scenario Outline blocks on the same tab.

Sometimes putting all your script blocks on 1 tab can be overwhelming. Or one of the flows is not yet ready for execution, so you don’t want to mix it with the others for export purposes. 

Figure 10 - New Automate script

You can create multiple tabs in Automate by clicking the + icon in the top ribbon (Figure 10 - 1), providing a unique name (Figure 10 - 2), selecting the type (Figure 10 - 3), and clicking “Create” (Figure 10 - 4).

When you click on any tab, you can click the pencil icon to edit the script's name, the trash icon to delete the script, or the copysheet icon to clone the entire tab. When you export into “Gherkin Feature File” or “Automated Test Framework”, you will have an option to select all scripts or only the specific ones.

Figure 11 - Automate script export

Manual

Manual (Figure 5 - 3) is a script editor in a grid format. It is aware of the parameters, values, and generated test cases of the current model. Manual makes it easy to create a .csv file with data-driven Action/Data/Expected Result sections - export will generate a script per Scenarios row, based on your Manual template structure.

Working on a Manual Script

The basic principles of working with a manual script are similar to those used in test management tools.

Top ribbon

Figure 12 - Top ribbon functions in Manual

Work on your step content in the editor grid (Figure 12 - 1). Important - to save the whole script, you need to use the “Save” button at the top (Figure 12 - 2). I.e. “Done” from each step does not auto-save the whole tab.

When you have unsaved changes, “Discard” button appears next to “Save”. It reverts the script to the last saved point.

Below the tabs, you can select one of the generated Scenario strengths and the specific Scenarios row to control your preview for the value replacement throughout the script (Figure 12 - 3). 

You can click the Usage button (Figure 12 - 4) for a refresher on the Manual Scripts functionality.

Step content

To define your steps, simply click the grey background in the row (e.g. between the words Action and Data) and provide the necessary text. The template format includes columns for Action, Data, and Expected Results. This selection is not customizable, but you only need to provide input in one of the three columns for a step to be saved. You can phrase the content of all 3 columns however you’d like. With that said, (in most cases) you must connect Scripts to the Scenarios table via the parameterized syntax.

Figure 13 - Editing then saving the step (lower part of the image is the state after clicking “Done” in the upper part)

Simply typing < will populate a list of autocomplete suggestions related to all the parameters in the model (Figure 13 - 1). For example, writing the step:

      User selects <Color> for the car's color

means that in each generated Manual script (in preview or export), the <Color> portion of the step will be replaced with the value of the Color parameter for the corresponding test case, such as:

      User selects Deep Blue Metallic for the car's color

      or

      User selects Red for the car’s color

      etc.

Parameterized syntax can be used in any of the 3 columns. The same parameter can be used multiple times in the same template. It is also possible to not use all the parameters in a given template. 

Once your step-level edits are complete, you can click “Cancel” or “Done” in the bottom right (Figure 13 - 2).

As you insert parameters into the script, they will become grayed out in the dropdown you get from typing <. It’s a quick way to check that you haven’t accidentally omitted any important variables by the time the script is finished.

Within a tab you can 

• copy individual cells within the tab or across tabs (Figure 13 - 3; e.g. hover over “Action” and click the “copy sheet” button that appears next to it, then paste elsewhere with Ctrl+V);
  
  
• clone the entire step (all 3 columns + conditions, if any) by clicking the “copy sheet” button located on the right side of each step (Figure 13 - 4);
  
  
• delete individual steps by clicking the “trash bin” button located on the right side of each step (Figure 13 - 6);
  
  
• reorder steps by dragging them (six-dot icon) or clicking corresponding arrows located on the left side of each step (Figure 13 - 5);
  
  
• hover over a white space around an existing step and click “New step” to insert it directly into the desired place (Figure 13 - 7).

Note: “Relax” feature has been retired, at least for now, so to recreate its behavior you would need to manually clone the steps and change parameter syntax & references.

Manual - Conditional Logic

Oftentimes application flows may have more or fewer steps that need to be executed for specific test cases. Step Conditions are the backbone of making sure your Manual Scripts accurately reflect the Scenarios. 

If a specific behavior doesn't apply to every test case, when you edit a step, you can click the “+” icon at the top, next to the word “Condition” (Figure 13 - 8). You can then specify the parameter value (or combination of them - just click “+” again) that would control whether the whole step (all 3 columns) will appear in the exported script for a given Scenarios row. 

We’ll use the same example as in the Automate section - in Tesla's car configurator, if the user is configuring a new Tesla Model S, there is no option for the Interior Layout. 

Figure 14 - Conditional steps in Manual Scripts

We set the condition to the <Model> parameter being not equal to “Model S” while editing the step (Figure 14 - 1). Then we click “Done”, and the color of the conditional step will reflect its applicability to the test case number you selected for preview (Figure 12 - 3):

• Green (Figure 14 - 2) means that this step will appear in the export for the particular test case (when that test has a Model different from “Model S”).
• Red with the crossed-out condition (Figure 14 - 3) means that this step will NOT appear in the export for the particular test case (when that test has “Model S”).

Note: for the avoidance of doubt, the color of the same step will change based on the preview - the image above just collates different states together, but (Figure 14 - 2) and (Figure 14 - 3) are mutually exclusive and won’t show as duplicated step entries.

By default, the condition type is set “is”, you can change it to “is not” by clicking on that text. If you added a condition by mistake and now need to specify that the step applies to all flows, you can click the trashbin icon next to the condition (Figure 14 - 4).

If you have the same Action that can have different, value-dependent expected results, you have to create a step per unique expected result, explicitly adding all condition combinations at the step level, then hardcoding the corresponding expected result in each step (see a few more tips in the Nuances section later). 

Keep in mind that you need to specify the minimum number of Values that will result in the step being applied or the expected result being triggered (i.e. if Expected Result XYZ appears for A1 + B1 + ! and A1 + B1 + #, and the third parameter only has those 2 values, then your Condition is only A1 + B1)

IMPORTANT -  Please make sure that you cover all the generated rows from the Scenarios screen across the Step Conditions, especially if you leverage multiple Manual tabs.

Manual - Multi-tab Approach

Each tab in Manual corresponds to 1 .csv file for transitions to certain test management tools and contains 1 script template. Each Manual script template can include as many steps as you need. Although, if you feel like the count becomes unmanageable, it can be a sign to split the script and/or the model.

Sometimes putting all your script variants on 1 tab can be overwhelming. Or one of the flows is not yet ready for execution, so you don’t want to mix it with the others for export purposes. 

Figure 15 - New Manual script tab

You can create multiple tabs in Manual by clicking the + icon in the top ribbon (Figure 15 - 1), providing a unique name (Figure 15 - 2), and clicking “Create” (Figure 15 - 3).

When you click on any tab, you can click the pencil icon to edit the script's name, the “trash bin” icon to delete the script, or the “copy sheet” icon to clone the entire tab. 

You cannot save an empty tab, so if you want to reset the script after the tab has been edited, you will need to delete the tab. 

When you export into a test management tool, you will have an option to select only the specific ones.

Figure 16 - Manual script export

You can now use the {scenario_name} name variable in the export of Manual scripts, it represents the tab name.

Note: there is an extra option, in the script selection dropdown, for exporters that support Manual Scripts. It’s called “Simple Parameters Script”, and it exports a step per parameter in the “Parameter Name = Parameter Value” format. It requires the Manual Scripts to be empty. If you just want to quickly move Scenarios into a test management tool for basic traceability, you can leverage this selection without actually touching Manual Scripts.

Implementation summary

The table below summarizes the broad use cases and their implementation in Manual or Automate Scripts:

Nuances

• To include one-off tests or a small set of negative testing scenarios or anything like that, you can intentionally avoid any parameterized syntax in the Scripts block (Manual or Automate->Scenario) and hardcode everything. That script will be disconnected from the model but will exist in a consolidated space with other scripts and will be exported.
  
  
• Keep in mind that, since Constraints support dependencies across any number of variables, you could add expected results to the Parameters screen, apply Constraints, then just pass the parameter name in the script without leveraging Conditional Logic described above. Depends on the specific rules involved and your comfort with Constraints.
  
  
• Make sure you understand the similarities and differences between Conditional steps with Expected Results in the Manual Scripts and Expected Outcomes in the Forced Interactions. There is a significant, yet subtle, difference:
  • Conditional steps with Expected Results in the Manual Scripts treat the Scenarios data table as a “read-only precondition” - if the combination in the Condition never happens for a scenario, the step will never be generated in export (will always be red in the preview).
  • Expected Outcomes in Forced Interactions are guaranteed to happen because their conditions (forced values) “edit” the Scenarios table.
    
    
    For instance, if you need to define an Expected Result that requires three or more specific values to appear in a single step (and you have generated a pairwise set of Scenarios), use the Forced Interactions feature (or a higher combination strength) to ensure the scenario is included in your suite. Then, use the Manual Scripts feature to document the Expected Result for export.
    
    If your Expected Result requires two or fewer specific values to appear in a single step (and you have generated a pairwise set of Scenarios), use the Manual Scripts Condition feature without additional prep work.

• There are situations where certain testing ideas did not make it into your test model. Often these ideas represent expected outcomes or validation steps and may be necessary to complete the test script.
  
  By using the <Parameter Name> syntax in the Scenario Outline block, but intentionally not using the name of a parameter in the test model, you can include <Some Additional Idea> and Automate will add an empty column to the Gherkin Examples data table.
  
  
• Although all generated rows on the Scenarios screen are unique, it's possible to include significantly fewer parameters in a Scenario or Scenario Outline block than are present in the model, and for this subset of parameters to result in duplicate selections from the data table rows. One use case is in integration testing, where all parameters are used in system A to create e.g. a mortgage application, but then the validation in system B only needs a subset. You want your scripts to be modular enough, so you create 1 Scenario block for system A and another block for system B.
  
  If you just include those selected parameters in an Automate script for system B, all duplicate combinations will be removed and the export counts will mismatch. To force every scenario to be included in the Scenario or Scenario Outline export, you can add the @enumerate_all Gherkin tag on the line above the words Scenario or Scenario Outline.