Testing: Integration/E2E/Regression rearchitecture spike

Created by: umpox

Motivation

Our integration / e2e / regression tests are difficult to develop and debug, and slow down developer productivity.

We looked into improving these tests by migrating from Mocha to Jest, so they would be closer to our unit tests. We ran into many issues that were complex to fix.

Some of the current problems include:

• Tests that are painful to debug
• Tests that are hard to read & write
  • Selecting elements is difficult and often requires className selectors
  • Quite a lot of boilerplate for a simple test
• Tests that use a different test runner
  • Most of are tests are written with Jest
  • These tests use Mocha
    • Different API, slower DX.
    • Less support, e.g. no working JUnit reporter meant we couldn’t integrate with Buildkite analytics
• Lack of browser support
  • Currently tests only run in Chrome
  • We’ve had some requests (e.g. from code insights) to support other browsers - like Safari.

Current setup

Browser environment Current: Puppeteer Preferred alternative - Playwright:

• Multi browser support
• Appears to be more active (e.g. Puppeteer vs Playwright pulse)

Test runner Current: Mocha Preferred alternative - Jest:

• More consistent + improved DX
• Better supported
• See Mocha vs Jest spike

Assertion library Current: Assert lib Preferred alternative - Jest:

• “expect()” API. More declarative. Also could be supporter with custom expect matchers that are specific to our integration tests

Element selector Current: Puppeteer API Preferred alternative - Playwright Testing Library:

• More consistent DX - we use React testing library in our unit tests
• Easier to read
• More brittle (less reliance on className selectors)

Network mocking (also recording) Current: PollyJS Preferred alternative - PollyJS (same?): Needs investigation, other options include:

• MSW
• Nock

Browser extension integration Current: Puppeteer Preferred alternative - Playwright:

• Supported on Chrome

Spike

Create a PR that uses the preferred alternative to look at:

• How feasible the migration would be
• Any problems we might run into
• Any other benefits/downsides that we might encounter.

Note: To ensure this spike is effective, we should ensure that:

• It includes at least 1 web test that uses network mocking
• It includes at least 1 browser extension test that uses network mocking

If feasible, the spike should result in the creation of a set of migration issues that we (or GitStart) are able to work on