Visual testing: add Percy
Created by: chrismwendt
This adds visual regression testing to our e2e tests. I was surprised by how easy this was to set up! This will help prevent bugs (see list at the bottom).
Links:
- Percy: https://percy.io/
- Percy + Puppeteer docs: https://docs.percy.io/docs/puppeteer
- Sample build: https://percy.io/Sourcegraph/Sourcegraph/builds/1522732
Visual testing process:
- In CI, we run
env PERCY_TOKEN=... percy exec -- yarn run test-e2e
, which starts the Percy agent that knows how to upload images to percy.io - In a test, we call
percySnapshot('Some screenshot name')
from@percy/puppeteer
- The Percy agent uploads all the images to percy.io
- The Buildkite build completes (visual regressions will not cause test failures in Buildkite, but will be added as another CI check requiring approval when there are differences)
- Percy checks the images:
- If there are no differences, the Percy GitHub CI check goes green
- If there are differences, the Percy GitHub CI check goes red, and someone must go to https://percy.io/Sourcegraph/Sourcegraph and approve the changes
Here's what it looks like:
UI bug list: