What you'll learn
- Benefits of data-driven automation with Test Case Designer
- Creation of compatible scripts in Playwright and Cypress
- Validation of data-driven Xray reports in Jira
Overview
Why data-driven automation can be a good choice
Definition
Data-driven testing is the practice of keeping the execution script separate from the test data. In other words, you will have a single automation script that will reference a data table containing multiple rows with scenarios.
When we talk about prioritizing tests for automation, it often comes down to high repetitiveness of actions and high number of iterations. Data-driven testing fits that pattern well and helps with some of the adoption barriers:
It removes the need to create an individual script for each test case, which is typically a time-consuming process. But often more importantly, it simplifies and accelerates maintenance because most changes to the test script do not affect the test data, and vice versa. Scaling execution up or down is also more flexible.
As an added bonus, if you need to switch frameworks/languages, your data sources would still be valid, and not as many scripts would need to be replaced.
- It can make collaboration easier since reviewing a data table does not require special technical skills. That, in turn, accelerates the overall process (since any subject matter expert can add iterations) and increases test data accuracy. The reporting can be set up to clearly reflect the status of each iteration, just as it would for separate scripts.
Of course, data-driven automation is not the universal solution, a few of its limitations are:
- Skill level requirements - if each iteration has enough “uniqueness” about its actions & validations or complexity of the expected result, trying to fit numerous iterations into 1 script will result in the overload of conditional statements. That would require a lot more automation experience.
- Shifting some of the problems to a different medium - manual creation & maintenance of data sources can be only marginally less cumbersome than those of individual scripts. Data relevancy and accuracy can still be large concerns.
For the first limitation, the collaborative analysis of the draft data tables should validate the pattern early on and properly choose the targets for data-driven automation, without excessively complicating the overall implementation. For the avoidance of doubt, you will likely always have a mix of automation approaches, where data-driven testing is one part.
For the second one, we can find a better alternative to manual data source management. For example, Xray Enterprise Test Case Designer (TCD).
Why data-driven automation with Test Case Designer can be an even better choice
- Faster data source creation - designing the table by hand can’t compete with the algorithmic generation when it comes to speed, especially for complex systems.
- More efficient maintenance - changes at the model level are automatically applied to all affected test cases.
- Minimized redundancy & maximized thoroughness - the TCD algorithm makes sure the data scenarios are as varied as possible while providing flexible coverage goals. That often leads to earlier detection of defects.
- Diverse collaboration artifacts - in addition to the data table, you can leverage auto-generated visualizations like mind map and coverage matrix to make sure all stakeholders are on the same page about the testing scope.
If at any point you need to switch to the “traditional” script-based approach, you can easily do it by leveraging the data-driven BDD/Robot Framework templates under Scripts. Create 1 Scenario/Test Case block inside TCD, export or sync it to as many issues as you have iterations. You don’t need to rebuild your models, and you still benefit from all 4 points above.
So let’s dive into how such optimized data-driven automation can be accomplished.
Prerequisites
For Playwright, in order to run this tutorial you need to have Nodejs and install the playwright-test runner.
For Cypress, you need to have Cypress installed.
How to do it
Common steps
We will discuss the implementation in 2 frameworks, with different complexity of the target apps. At a high level, steps are similar for each example:
- Create the setup in Jira with the requirement/story and the Xray Test Plan.
- Create a model for the target system in TCD.
- can add the model URL to the requirement story.
- Export the data table as CSV.
- Create the data-driven execution script in the selected framework/language.
- Run that code and generate the report format that is supported by Xray.
- optionally, configure the level of evidence in the report and add links to the requirement and/or test plan from step 1.
- Import the report file via REST API (directly or through a CI/CD tool).
- Enjoy full visibility of the results in Xray.
Integrate Test Case Designer with Playwright
For Playwright versions 1.34 and later, if you want to attach screenshots, you will need to additionally install this Playwright reporter and change the config file or the command line statement accordingly.
The scope of this tutorial is:
- passing different combinations of data to the calculator app at https://treasurydirect.gov/BC/SBCGrw
- validating that the evaluation results are returned successfully (by checking the presence of the title on the page, since we don't know the actual formula behind the app)
Build a TCD model and export scenarios
The model in TCD looks like this (a fairly straightforward approach of a parameter per UI field):
Note that:
- Shorter parameter names without spaces or special characters work better.
- The spelling of value names should match the syntax expected by the automation script. You can share your draft TCD model with the engineers to verify that.
- For data table and Scripts->Automate exports, only value expansion text will be populated, so we can still benefit from the equivalence class approach for, e.g., IntRate or FedTaxRate, without worrying about the value name syntax.
After generating Scenarios, we can export the CSV data table from the same screen:
As the automation is being developed, it may make sense to leverage the “1-way interactions” goal for the TCD algorithm. It exposes the execution to each possible value with the smallest number of iterations, which helps with the feedback speed.
You can import the model (Server/DC ; Cloud) and try this process yourselves:
Create a script in Playwright
On the Playwright side, we first adjust the configuration file to support Xray options with the appropriate Junit report formatting (“const xrayOptions” and “reporter” sections):
Then implement the driver script:
The core detail about this approach is, using the extra libraries, we first read the csv from TCD, then iterate through it with the “for” loop:
...
import fs from 'fs';
import path from 'path';
import { parse } from 'csv-parse/sync';
const records = parse(fs.readFileSync(path.join(__dirname, 'TCDDemo.csv')), {
columns: true,
skip_empty_lines: true
});
for (const record of records) {
...
};
The parameters (aka column names from the CSV, e.g. record.InitialInvAmount) are passed in typical Playwright methods (like Fill or selectOption) instead of hardcoding text.
To improve the clarity and traceability, we can add a link to the requirement story, parameterized name & description, and custom screenshots.
Import results to Xray
Once the script completes the execution, we can import the results via, e.g., curl (parts in {} should be customized based on your details):
curl -H "Content-Type: multipart/form-data" -u {username}:{password} -F "file=@{ReportName}.xml" “https://{JiraURL}/rest/raven/1.0/import/execution/junit?projectKey={ProjectKey}&testPlanKey={TestPlanIssueKey}”
As a result, you should see five Test issues (linked to the requirement story and to the Test Plan issue) and a Test Execution issue (with the screenshots under Execution details of each run):
Integrate Test Case Designer with Cypress
The scope of this tutorial is:
- providing different combinations of applicant data to the web mortgage quoting system at https://www.wellsfargo.com/mortgage/calculators/.
- The flow starts with selecting “Buy a home” as the purpose, then entering location, amount, and credit rating details. After the initial results, you can adjust the term.
- checking the app's ability to return the loan evaluation results successfully by asserting the page title (without looking at the specific numbers).
Build a TCD model and export scenarios
The model in TCD looks like this:
Note that:
- It’s not wrong to include 1-value parameters for test data rationale (like “Purpose”) if you feel like it’s better than hardcoding them in the script (often because you expect that parameter to grow over time).
- For some web apps, selection by labels can be too complicated, so you may have to switch to indices (e.g. “Term” values). Refer to the feedback from your automation team.
- This example showcases how to deal with conditional steps.
“If the DownpaymentPercentage is 20 or higher, then an additional question about the number of borrowers doesn’t appear". On the TCD side, you can have:- either a “special” value (like “NABorrowers” for “NumberofPeople”) with the corresponding set of constraints
- or a “skip” constraint (DownpaymentPercentage[20,21 - 50] >> NumberofPeople)
The choice will determine which trigger will be assigned to the automation step that we will show later.
- either a “special” value (like “NABorrowers” for “NumberofPeople”) with the corresponding set of constraints
You can import the model and try this process yourselves:
Create a script in Cypress
On the Cypress side, you will also first need to adjust the package and the config files to support csv handling with 3 imports:
The optimized data table can go into cypress/fixtures while the execution script in cypress/e2e would look like this:
Similarly to the Playwright example, we will first read the CSV file, then iterate the script through it with the “forEach” loop.
const tcdloandata = Cypress.env('TCDDemo')
...
tcdloandata.forEach((l) => {
...
})
Keep in mind that the final Test Issue naming convention in Xray will concatenate these lines:
describe(`Quote a mortgage`, () => {
...
it(`With ${l.PurchasePrice} purchase price and ${l.DownpaymentPercentage} downpayment for ${l.NumberofPeople}`, () => {
...
})
})
So you need to split the desired summary accordingly while parameterizing the second line.
The code below is responsible for the conditional logic we mentioned earlier around Downpayment Percentage and Number of Borrowers.
...
cy.get("body").then($body => {
if ($body.find(`#${l.NumberofPeople}`).length > 0) {
cy.get(`#${l.NumberofPeople}`).then($btn => {
if ($btn.is(':visible')){
cy.wrap($btn).click({force: true})
} else {
cy.log('Question about borrowers did not appear')
}
});
} else {
cy.log('Question about borrowers did not exist on the page')
assert.isOk('everything','everything is OK');
}
})
...
It searches for the dynamically defined element (`#${loan.NumberofPeople}`), checks whether it’s present and visible, and, if yes, clicks it, otherwise, logs a message. In this implementation of the TCD model, ‘#NABorrowers’ element would never be found, so the automation skips the step whenever that value comes from the CSV.
- If you know the exact behavior of the element (visibility vs presence), that conditional logic can be simplified.
- If the “skip” constraint route is taken, the same code could work. Or you may need to adjust the dynamic element syntax if ‘#_‘ is treated as undefined/syntax error.
Import results to Xray
The results import process would be identical to the Playwright one. You will notice that 2 tests are failing as the results are not returned:
This is a decent example of the execution feedback loop for combinatorial test design: we will often exercise the combinations that haven't been explicitly mentioned in requirements. Since all individual values are within valid ranges (based on UI tooltips), we would review the scenario as a whole with the stakeholders and confirm the expected behavior. (You can notice that both failures share at least 1 similarity: the purchase price is close to the upper limit.)
If the lack of results is valid, we can make the assertion conditional on data in each iteration or adjust the values/constraints in the TCD model to focus more on the combinations that return rates.
Tips
- Generating CSV from the Scenarios screen in TCD results in only value expansion names being included, where applicable. The CSV option on the Export screen is geared more towards manual execution and will produce "Value name - Value expansion name" syntax.
- To create script steps in Playwright, if you are not yet familiar with the syntax, you can use the CodeGen feature.
- The csv processing for Cypress recommends neat-csv package. Depending on the version, you need to switch the Cypress config and package files between CommonJS and ES Module scripting. Our example uses ES Module. Or you can try an alternative package.
References
- https://playwright.dev/docs/test-reporters#junit-reporter
- https://playwright.dev/docs/test-parameterize#create-tests-via-a-csv-file
- https://docs.cypress.io/examples/recipes
- https://learn.cypress.io/advanced-cypress-concepts/using-data-to-build-dynamic-tests
- https://github.com/cypress-io/cypress-example-recipes/tree/master/examples/fundamentals__dynamic-tests-from-csv