Why data-driven automation can be a good choice

When we talk about prioritizing tests for automation, it often comes down to high repetitiveness of actions and high number of iterations. Data-driven testing fits that pattern well and helps with some of the adoption barriers:

1. It removes the need to create an individual script for each test case, which is typically a time-consuming process. But often more importantly, it simplifies and accelerates maintenance because most changes to the test script do not affect the test data, and vice versa. Scaling execution up or down is also more flexible.
   

   Info
   As an added bonus, if you need to switch frameworks/languages, your data sources would still be valid, and not as many scripts would need to be replaced.

2. It can make collaboration easier since reviewing a data table does not require special technical skills. That, in turn, accelerates the overall process (since any subject matter expert can add iterations) and increases test data accuracy. The reporting can be set up to clearly reflect the status of each iteration, just as it would for separate scripts.

Of course, data-driven automation is not the universal solution, a few of its limitations are:

1. Skill level requirements - if each iteration has enough “uniqueness” about its actions & validations or complexity of the expected result, trying to fit numerous iterations into 1 script will result in the overload of conditional statements. That would require a lot more automation experience.
   
   
2. Shifting some of the problems to a different medium - manual creation & maintenance of data sources can be only marginally less cumbersome than those of individual scripts. Data relevancy and accuracy can still be large concerns.

For the first limitation, the collaborative analysis of the draft data tables should validate the pattern early on and properly choose the targets for data-driven automation, without excessively complicating the overall implementation. For the avoidance of doubt, you will likely always have a mix of automation approaches, where data-driven testing is one part.

For the second one, we can find a better alternative to manual data source management. For example, Xray Test Case Designer (TCD). 

Why data-driven automation with TCD can be an even better choice

• Faster data source creation - designing the table by hand can’t compete with the algorithmic generation when it comes to speed, especially for complex systems.
  
  
• More efficient maintenance - changes at the model level are automatically applied to all affected test cases.
  
  
• Minimized redundancy & maximized thoroughness - the TCD algorithm makes sure the data scenarios are as varied as possible while providing flexible coverage goals. That often leads to earlier detection of defects.
  
  
• Diverse collaboration artifacts - in addition to the data table, you can leverage auto-generated visualizations like mind map and coverage matrix to make sure all stakeholders are on the same page about the testing scope.

So let’s dive into how such optimized data-driven automation can be accomplished.

How to do it

Common steps

We will discuss the implementation in 2 frameworks, with different complexity of the target apps. At a high level, steps are similar for each example:

1. Create the setup in Jira with the requirement/story and the Xray Test Plan.
2. Create a model for the target system in TCD.
1. can add the model URL to the requirement story.
3. Export the data table as CSV.
4. Create the data-driven execution script in the selected framework/language.
5. Run that code and generate the report format that is supported by Xray.
1. optionally, configure the level of evidence in the report and add links to the requirement and/or test plan from step 1.
6. Import the report file via REST API (directly or through a CI/CD tool).
7. Enjoy full visibility of the results in Xray.

Playwright implementation

The scope of this tutorial is:

• passing different combinations of data to the calculator app at https://treasurydirect.gov/BC/SBCGrw
  
• validating that the evaluation results are returned successfully (by checking the presence of the title on the page, since we don't know the actual formula behind the app)

The model in TCD looks like this (a fairly straightforward approach of a parameter per UI field):

Image Modified

Note that:

• Shorter parameter names without spaces or special characters work better.
• The spelling of value names should match the syntax expected by the automation script. You can share your draft TCD model with the engineers to verify that.
  
• For data table and Scripts->Automate exports, only value expansion text will be populated, so we can still benefit from the equivalence class approach for, e.g., IntRate or FedTaxRate, without worrying about the value name syntax.

After generating Scenarios, we can export the CSV data table from the same screen:

As the automation is being developed, it may make sense to leverage the “1-way interactions” goal for the TCD algorithm. It exposes the execution to each possible value with the smallest number of iterations, which helps with the feedback speed.

You can import the model and try this process yourselves:

On the Playwright side, we first adjust the configuration file to support Xray options with the appropriate Junit report formatting (“const xrayOptions” and “reporter” sections):