Skip to content

Add regression tests for onboarding flow

Administrator requested to merge bl/regression-onboarding into master

Created by: beyang

Adds onboarding regression tests (onboarding.test.ts). These tests are a little bit different in that they print out manual verification steps at the very end:

              @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
              @@@@ Manual verification steps required!!! @@@@
              @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

              Please verify the following screenshots match the corresponding descriptions:

              initial-progress-bar-is-gray-circle.png:  "gray circle"
              confetti-appears-after-first-search.png:  "confetti"
              progress-bar-after-initial-search-is-half-green.png:      "50% green circle"
              confetti-appears-after-find-refs.png:     "confetti"
              progress-bar-after-search-and-fnd-refs-is-full-green.png: "100% green circle"

This is because there are certain visual things to verify that are hard to encode in logic, but easy for a human to eyeball.

Side note: I considered using Percy, but image-diffing doesn't provide good signal here (consider the problem of diffing a screenshot of the confetti animation, which can vary based on exactly when you take the screenshot). It was easier just to take a screenshot and ask the human to verify at the end. I'm a little worried that the message will be ignored, so am considering whether to require an explicit "YES I verified this" prompt at the end of the test suite. Thoughts on that?

There were a few other minor refactors such as renaming some utility functions, moving some error handling, and adding extra options to utility functions.

This only tests onboarding for non-admin users. I still need to add another tests for admin onboarding, which includes additional steps. It also does not add tests to verify that the onboarding status menu appears, which I'll do in a later PR.

Merge request reports