---
title: "Testing web applications using Applitools Eyes"
canonical: "https://docs.getxray.app/space/XRAY/301676170/Testing%20web%20applications%20using%20Applitools%20Eyes"
format: markdown
---
> ℹ️ **What you'll learn**
> ℹ️ 
> ℹ️ - Define tests using Playwright-test
> ℹ️ - Add visual validations with Applitools Eyes
> ℹ️ - Run the test and push the test report to Xray
> ℹ️ - Validate in Jira that the test results are available

> ⚠️ **Source-code for this tutorial**
> ⚠️ 
> ⚠️ - code is available in [GitHub](https://github.com/Xray-App/tutorial-js-playwright-Applitools-Eyes)

# Overview

Playwright is a recent browser automation tool that provides an alternative to Selenium.

Applitools Eyes is a visual AI test automation tool that have an SDK available that you can add to your test project allowing visual validations.


---

# Prerequisites


<details>
<summary>Expand</summary>

For this example we will use[ Playwright Test Runner](https://github.com/microsoft/playwright-test/blob/master/README.md) and [Applitools Eyes SDK](https://applitools.com/docs/topics/sdk/sdk.html).


 We will need:

- Access to a [demo site](https://xray-essentials-petclinic.herokuapp.com/) that we aim to test
- Node.js environment with Playwright and [Playwright Test Runner](https://github.com/microsoft/playwright-test/blob/master/README.md)
- [Applitools Eyes SDK](https://applitools.com/docs/api/eyes-sdk/index-gen/classindex-playwright-javascript.html)
</details>


To start using the [Playwright Test Runner](https://github.com/microsoft/playwright-test/blob/master/README.md) please follow the [Get Started](https://github.com/microsoft/playwright-test#get-started) documentation.

The tests consist in validating 3 features of the [demo site](https://xray-essentials-petclinic.herokuapp.com/): Home link, Find owners functionality and veterinarians link.

We want to add visual validations to these tests, so we have included the Applitools Eyes SDK to be able to use the comparison abilities of the tool.

Before coding the tests start by registering in the Applitools Eyes site and obtain an API-KEY (that is what we will use in the test execution to ship screenshots to the tool for comparison), more information on how to do it [here](https://applitools.com/docs/topics/overview/obtain-api-key.html?Highlight=api%20key).



We started by defining *PageObjects* that will represent the pages we will interact with, we have defined three, as we see below:

<details>
<summary>./models/owners.js</summary>

```javascript
const config = require ("../config/config.json");

class OwnersPage {

    constructor(page) {
      this.page = page;
    }

    async navigate() {
      await this.page.goto(config.endpoint);
    }
    
    async click_find_owners_button(){
        await this.page.click(config.find_owners_button);
    }

  }
  module.exports = { OwnersPage };
```
</details>

<details>
<summary>./models/home.js</summary>

```javascript
const config = require ("../config/config.json");

class HomePage {

    constructor(page) {
      this.page = page;
    }

    async navigate() {
      await this.page.goto(config.endpoint);
    }
    
    async getMenuEntry(){
        return await this.page.locator(config.top_menu_entry).first();
    }

    async getHomeText(){
        return config.home_text;
    }

  }
  module.exports = { HomePage };
```
</details>

<details>
<summary>./models/veterinarians.js</summary>

```javascript
const config = require ("../config/config.json");

class VetsPage {

    constructor(page) {
      this.page = page;
    }

    async navigate() {
      await this.page.goto(config.endpoint);
    }
    
    async getTopMenuEntry(){
        return this.page.locator(config.vet_menu_entry).first();
    }

    async getVetsText(){
        return config.vet_text;
    }

  }
  module.exports = { VetsPage };
```
</details>


Plus a configuration file where we have the identifiers that will match the elements in the page, this will add an extra abstraction layer to the tests allowing us to redefine locators or text without changing the code.

<details>
<summary>config.json</summary>

```javascript
{
    "endpoint" : "https://xray-essentials-petclinic.herokuapp.com/",
    "owners_link" : "a[title=\"find owners\"]",
    "top_menu_entry" :"//*[@id=\"main-navbar\"]/ul/li[1]/a",
    "vet_menu_entry" : "//*[@id=\"main-navbar\"]/ul/li[3]/a", 
    "find_owners_button" : "a[title=\"find owners\"]",
    "vet_text" : "Veterinarians",
    "home_text" : "Home"
}
```
</details>


We added an helper file that will parse returned information and add valuable information returned by Applitools Eyes to the Junit report.

<details>
<summary>helper.js</summary>

```javascript
class Helper {

  constructor() {
    
  }

  handleTestResults(summary){
    let ex = summary.getException();
    if (ex != null ) {
        console.log("System error occurred while checking target.\n");
    }
    let result = summary.getTestResults();
    if (result == null) {
        console.log("No test results information available\n");
    } else {
        console.log("[Eyes URL|%s] \\\\ AppName = %s \\\\ testname = %s \\\\ status = %s \\\\ different = %s \\\\ Browser = %s \\\\ OS = %s \\\\ viewport = %dx%d \\\\ matched = %d \\\\ mismatched = %d \\\\ missing = %d\\\\ aborted = %s\\\\",
            result.getUrl(),
            result.getAppName(),
            result.getName(),
            result.getStatus(),
            result.getIsDifferent(),
            result.getHostApp(),
            result.getHostOS(),
            result.getHostDisplaySize().getWidth(),
            result.getHostDisplaySize().getHeight(),
            result.getMatches(),
            result.getMismatches(),
            result.getMissing(),
            (result.getIsAborted() ? "aborted" : "no"));
            let steps = result.getStepsInfo();
            steps.forEach(step => {
              console.log("StepName = %s, different = %s\\\\", step.getName(), step.getIsDifferent());
            });
        }

  };

  }
  module.exports = { Helper };
```
</details>


The tests that validate if the features are behaving as expected are below, notice that we are using the Applitools Eyes SDK and adding checks on the tests in the moments we want to have visual validations.

For the tutorial purpose we will focus in the *Owners *validations (the others will be similar with more or less actions). 

<details>
<summary>login.spec.ts</summary>

```javascript
import { test, expect } from '@playwright/test';
import { OwnersPage } from "../models/owners";
import { Helper } from "../models/helper"
const { Eyes, ClassicRunner, Target , Configuration, BatchInfo, MatchLevel, TestResultContainer, TestResults} = require('@applitools/eyes-playwright')

test.describe("PetClinic validations", () => {
  let eyes, runner;//, default_url;

  test.beforeEach(async () => {

    // Initialize the Runner for your test.
    runner = new ClassicRunner();

    // Create Eyes object with the runner
    eyes = new Eyes(runner);

    // Initialize the eyes configuration
    const configuration = new Configuration();

    // create a new batch info instance and set it to the configuration
    configuration.setBatch(new BatchInfo('PetClinic Batch - Playwright - Classic'));

    // Define the match level we need for our tests
    eyes.setMatchLevel(MatchLevel.Strict);

    // Set the configuration to eyes
    eyes.setConfiguration(configuration);

  });

  test('Validate find owners link', async ({ page }) => {
    const ownersPage = new OwnersPage(page);
    await ownersPage.navigate();
    await eyes.open(page, 'PetClinic', 'FindOwnersLink', { width: 800, height: 600 });
    await ownersPage.click_find_owners_button();
    await eyes.check(Target.window().fully());
    await eyes.close();
  });

  test.afterEach(async () => {
    const helper = new Helper();
    // If the test was aborted before eyes.close was called, ends the test as aborted.
    await eyes.abort();

    // We pass false to this method to suppress the exception that is thrown if we
    // find visual differences
    const results = await runner.getAllTestResults(false);

    results.getAllResults().forEach(result  => {
      helper.handleTestResults(result);
    });
    
  });
}) 
```
</details>


Looking to this class in more details we can see different areas: 

- *test.beforeEach()*
- *test()*
- *test.afterEach()*

In the "*test.beforeEach*" we are configuring the runner that will be used by the Eyes instance, as you can see, we are using the classic one (Eyes have another available called "*<span style="color: #172B4D">Visual Grid Runner</span>*<span style="color: #172B4D">" that interacts with the Eyes Ultrafast Grid server to render the checkpoint images in the cloud). </span>

<span style="color: #172B4D">We defined a configuration object that will hold the configuration for the instance, we are defining the Batch named 'PetClinic Batch - Playwright - Classic', defining the match level (in our case we are using the recommended one </span>*<span style="color: #172B4D">Strict</span>*<span style="color: #172B4D"> but there are </span>[<span style="color: #172B4D">more</span>](https://applitools.com/docs/common/cmn-eyes-match-levels.html)<span style="color: #172B4D"> available).</span>

<span style="color: #172B4D">In the test itself we have a normal Playwright test with additions from the Applitools Eyes SDK, let's look into those in more detail:</span>

- *<span style="color: #172B4D">await eyes.open(page, 'PetClinic', 'FindOwnersLink', { width: 800, height: 600 });</span>*<span style="color: #172B4D"> - </span><span style="color: #2c2c2c">to start a test, before calling any of the check methods and we are defining the AppName, TestName and ViewPortSize.</span>
- *<span style="color: #172B4D">await eyes.check(Target.window().fully());</span>*<span style="color: #172B4D"> - </span><span style="color: #2c2c2c">Run a checkpoint. Uses Fluent arguments to specify the various </span>[<span style="color: #2c2c2c">parameters</span>](https://applitools.com/docs/api/eyes-sdk/classes-gen/class_eyes/method-eyes-check-playwright-javascript.html)<span style="color: #2c2c2c">.</span>
- *<span style="color: #172B4D">await eyes.close();</span>*<span style="color: #172B4D"> - </span><span style="color: #2c2c2c">Call this method at the end of the test. This terminates the sequence of checkpoints, and then waits synchronously for the test results and returns them.</span>

Finally in the "*test.afterEach*" we make sure to close all eyes instances by calling the "abort" method and process the results that are returned by the runner.


Once the code is implemented, we will run it to define the baseline (<span style="color: #2c2c2c">A</span><span style="color: #2c2c2c"> </span><span style="color: #2c2c2c">baseline</span><span style="color: #2c2c2c"> </span><span style="color: #2c2c2c">stores a sequence of reference images</span>), that will be used to compare to the next tests. We achieve that with the following command:

```shell
APPLITOOLS_API_KEY="API_KEY" PLAYWRIGHT_JUNIT_OUTPUT_NAME=results.xml npx playwright test ./tests/* --browser=chromium --reporter=junit,line
```

> ⚠️ If the APPLITOOLS_API_KEY is not defined the tests will be executed but the screenshots will not be sent to Applitools Eyes.


The output generated shows how many tests have been executed, produces a Junit report and returns the link to check the visual assertions.

![image](media://b2bd37f6-2bf5-483b-a661-fa3d4a9ebdd4)


In Applitools Eyes interface we can see that a new application was created with 3 tests:

![image](media://d77f8f62-1b88-40bd-ae9e-9071620e925e)

If we navigate to the test results we will see the three tests properly named, information about the OS, Browser and Viewport used, a screenshot taken and the notion if is new or not and a date.

![image](media://9016e520-eb80-4560-8ed7-935692bcd760)

At this point we have generated our baseline and the tests are behaving as expected, we will now introduce a change in the application and remove strings from the Owners test that will make the test to succeed but the visual validation will fail as it will not match the baseline and thus failing the tests overall.

After the second execution the output terminal will have the following information:

![image](media://1af6da9a-83e7-4054-9281-02d7dd8b6c20)

The report generated will contain the following information:

<details>
<summary>Junit Report</summary>

```
<testsuites id="" name="" tests="3" failures="1" skipped="0" errors="0" time="23.226">
<testsuite name="tests/home.spec.ts" timestamp="1639395430782" hostname="" tests="1" failures="0" skipped="0" time="14.955" errors="0">
<testcase name="PetClinic validations Validate home link" classname="[chromium] › tests/home.spec.ts:32:3 › PetClinic validations › Validate home link" time="14.955">
<system-out>
[Eyes URL|https://eyes.applitools.com/app/batches/00000251762905362858/00000251762905362326?accountId=o9A_TwFAGkSW8d9i1ZlDBg~~] \\ AppName = PetClinic \\ testname = HomeLink \\ status = Passed \\ different = false \\ Browser = Chrome 97.0 \\ OS = Mac OS X 10.15 \\ viewport = 800x600 \\ matched = 1 \\ mismatched = 0 \\ missing = 0\\ aborted = no\\
StepName = , different = false\\

</system-out>
</testcase>
</testsuite>
<testsuite name="tests/owners.spec.ts" timestamp="1639395430782" hostname="" tests="1" failures="1" skipped="0" time="21.788" errors="0">
<testcase name="PetClinic validations Validate find owners link" classname="[chromium] › tests/owners.spec.ts:31:3 › PetClinic validations › Validate find owners link" time="21.788">
<failure message="owners.spec.ts:31:3 Validate find owners link" type="FAILURE">
  [chromium] › tests/owners.spec.ts:31:3 › PetClinic validations › Validate find owners link =======

    Error: Test 'FindOwnersLink' of 'PetClinic' detected differences! See details at: https://eyes.applitools.com/app/batches/00000251762905361920/00000251762905361545?accountId=o9A_TwFAGkSW8d9i1ZlDBg~~

      35 |     await ownersPage.click_find_owners_button();
      36 |     await eyes.check(Target.window().fully());
    > 37 |     await eyes.close();
         |     ^
      38 |   });
      39 |
      40 |   test.afterEach(async () => {

        at Eyes.close (/Users/cristianocunha/Documents/Projects/applitoolseyes/node_modules/@applitools/eyes-api/dist/Eyes.js:247:23)
        at processTicksAndRejections (internal/process/task_queues.js:93:5)
        at /Users/cristianocunha/Documents/Projects/applitoolseyes/tests/tests/owners.spec.ts:37:5
        at WorkerRunner._runTestWithBeforeHooks (/Users/cristianocunha/Documents/Projects/applitoolseyes/node_modules/@playwright/test/lib/workerRunner.js:478:7)

</failure>
<system-out>
[Eyes URL|https://eyes.applitools.com/app/batches/00000251762905361920/00000251762905361545?accountId=o9A_TwFAGkSW8d9i1ZlDBg~~] \\ AppName = PetClinic \\ testname = FindOwnersLink \\ status = Unresolved \\ different = true \\ Browser = Chrome 97.0 \\ OS = Mac OS X 10.15 \\ viewport = 800x600 \\ matched = 0 \\ mismatched = 1 \\ missing = 0\\ aborted = no\\
StepName = , different = true\\

</system-out>
</testcase>
</testsuite>
<testsuite name="tests/veterinarians.spec.ts" timestamp="1639395430782" hostname="" tests="1" failures="0" skipped="0" time="15.489" errors="0">
<testcase name="PetClinic validations Validate veterinarians link" classname="[chromium] › tests/veterinarians.spec.ts:31:3 › PetClinic validations › Validate veterinarians link" time="15.489">
<system-out>
[Eyes URL|https://eyes.applitools.com/app/batches/00000251762905363795/00000251762905363202?accountId=o9A_TwFAGkSW8d9i1ZlDBg~~] \\ AppName = PetClinic \\ testname = VetsLink \\ status = Passed \\ different = false \\ Browser = Chrome 97.0 \\ OS = Mac OS X 10.15 \\ viewport = 800x600 \\ matched = 1 \\ mismatched = 0 \\ missing = 0\\ aborted = no\\
StepName = , different = false\\

</system-out>
</testcase>
</testsuite>
</testsuites>
```
</details>

When we access the link provided by Applitools Eyes we can see the visual changes detected:

![image](media://199cb1ad-ccb2-4076-9438-1b59e32c4ef8)

When accessing the details we can see the actual differences detected between the baseline and the latest test side by side:

![image](media://d2b1007d-f8cd-4cec-89e7-aa8fb090bf41)


Notes:

- Applitools will let you analyse further the problem by filtering the view in different layers.
- Applitools have available the possibility to define regions, regions to be ignore for example if the content is too dynamic.
- Applitools let you add annotations, such as remarks or setting a bug.
- By default Playwright will execute tests for the 3 browser types available (that is why we are forcing to execute only for one browser)
- By default all the tests will be executed in headless mode
- Folio command line will search and execute all tests in the format: *"**/?(*.)+(spec|test).[jt]s" *
- In order to get the Junit test report please follow this [section](https://github.com/microsoft/playwright-test/blob/master/README.md#export-junit-report)


---

# Integrating with Xray

<span style="color: #172B4D">As we saw in the above example, where we are producing Junit reports with the result of the tests, it is now a matter of importing those results to your Jira instance, this can be done by simply submitting automation results to Xray through the REST API, by using one of the available CI/CD plugins (e.g. for Jenkins) or using the Jira interface to do so.</span>


> Macro (ui-tabs)
> 
> > Macro (legacy-content)
> 
> 
> 
> > Macro (legacy-content)


# Tips

- <span style="color: #000000">after results are imported, in Jira Tests can be linked to existing requirements/user stories, so you can track the impacts on their coverage.</span>
- <span style="color: #000000">results from multiple builds can be linked to an existing Test Plan, to facilitate the analysis of test result trends across builds.</span>
- <span style="color: #000000">results can be associated with a Test Environment, in case you want to analyze coverage and test results by that environment later on. A Test Environment can be a testing stage (e.g. dev, staging, prepod, prod) or a identifier of the device/application used to interact with the system (e.g. browser, mobile OS).</span>



---

# References

- [https://github.com/microsoft/playwright-test/blob/master/README.md](https://github.com/microsoft/playwright-test/blob/master/README.md)
- [https://playwright.dev/](https://playwright.dev/)
- [https://playwright.tech/blog/using-jest-with-playwright](https://playwright.tech/blog/using-jest-with-playwright)
- [https://applitools.com/docs/index.html](https://applitools.com/docs/index.html)
- [https://applitools.com/docs/topics/sdk/sdk.html](https://applitools.com/docs/topics/sdk/sdk.html)

> Macro (toc)

> Macro (style)