parallelization

Overview

We recently held our Spring Summit consisting of six workshops hosted by each of our practice areas. On March 17th, 2023, our SDET Practice led a series of talks and workshops on Microsoft’s Playwright. Playwright is a tool that enables end-to-end testing of modern web applications. Playwright works with all modern web browsers including: Chromium, Firefox, and WebKit. For more information see: https://playwright.dev/

Playwright Parallelism

Playwright achieves parallelism by running multiple simultaneous processes threads. Each thread runs one Playwright Test File at a time. That is to say, if we have 10 test files and 5 threads, then the Playwright orchestrator will send a test file to each thread (or process), thus running five test files in parallel. As a thread becomes available, it will be provided as part of the next test file.

Tests in a test file are run in order by default. However, it is possible to parallelize those as well; keep in mind that tests run in parallel are in independent processes and cannot communicate with each other. So if one test is dependent on another, you would not want to run them in parallel.

Disabling Parallelism

You may on occasion have a good reason to disable parallel testing. For instance you may be targeting hardware that has limited process threads, memory, or CPU. Alternatively, there may be a particular workflow you need to follow and it would be best to run in a particular sequence.

Fortunately, Playwright does allow you to disable parallelism by setting the number of workers to one in the `test.describe.configure` settings. You can also do this from the command line runner with `npx playwright test –workers=1`

Control Test Order

By default the tests in a Playwright Test File are run in order of declaration. However, you can override that in some circumstances. For instance if you have disabled parallelism, test files will be run in alphabetical order by default. So you could control the order by coming with a good naming convention.

You can go a step further by controlling the order in helper functions. For more information on that, we recommend that you see the documentation located at: https://playwright.dev/docs/test-parallel#use-a-test-list-file

Playwright Reporting

Playwright Test includes the ability to create reports based on test results. Playwright includes seven predefined reporters and provides support for creating custom reporters.

The Predefined Reporters Are:

• List Reporter

Prints a line for each test being run, and is the default for all executions except when running with CI. For CI the Dot reporter is the default.

Example Output:

• Line Reporter

According to the official documentation, a Line Reporter is more concise than the List Reporter. It uses a single line to report the last finished test, and prints failures when they occur.

Line Reporter is useful for large test suites where it shows the progress but does not spam the output by listing all the tests.

Example Output:

• Dot Reporter

The Dot Reporter provides a concise report by just producing a single character per successful test. The Dot Reporter is the default reporting method for CI. This method can be very useful anytime you do not want a lot of output.

Example Output:

• HTML Reporter

As its name suggests, the HTML reporter generates an HTML based report. The HTML Reporter according to the documentation, by default, HTML reports are opened automatically if some of the tests failed. You can control this behavior via the open property in the Playwright config. The possible values for that property are always, never and on-failure (default).

You can also configure host and port that are used to serve the HTML Report.

• JSON Reporter

This reporter generates results in JSON (JavaScript Object Notation). You may find it useful to have this written to a file rather than the screen. 

Below is an example of how to do this from Bash:

Alternatively you can define this behavior in the configuration file:

• JUnit Reporter

This will produce a report in JUnit-style xml. As with the JSON Reporter, you will likely want to write this out to a file. The JUnit Reporter offers support for a robust set of options. If you are testing an app written in Java that utilizes JUnit for tests, this reporter can be highly useful. 

More information about this can be found at: https://playwright.dev/docs/test-reporters#junit-reporter

• GitHub Reporter

Provides automatic failure annotations when running in GitHub actions. The other reporters work with GitHub Actions, but do not provide annotations.

Invoke A Reporter

It is quite easy to specify which reporter to use, by setting the name like in the example below:

Custom Reporters

As previously mentioned, you can create a custom reporter. Playwright provides an API to help with this at https://playwright.dev/docs/api/class-reporter. 

Below is an example of a custom TypeScript reporter courtesy of the Playwright website:

As with the pre-defined reporters, you can use your custom ones by specifying the name of the reporter. For instance, the custom reporter above can be accessed as: