Take percy snapshots of storybooks

Created by: felixfbecker

Closes #9565 Closes #10631

Also simplifies our Storybook config (Storybook has new recommended config approaches since 5.3).

The coverage and uploading to Percy is separate, because Percy doesn't actually run our storybook in CI. It just gets all static HTML and uploads it to Percy to do the screenshotting in the cloud. We therefor track the coverage by loading each story in Puppeteer in a separate job.

This fixes one important bug that took a long time to find - our Puppeteer coverage reporter overwrote coverage files from previous tests. This must have been a problem for e2e coverage too.

Check out the Percy report in the GitHub status checks below, and the coverage report in the Codecov comment below.

Created by: codecov[bot]

Codecov Report

Merging #10622 into master will increase coverage by 0.12%. The diff coverage is 10.00%.

@@            Coverage Diff             @@
##           master   #10622      +/-   ##
==========================================
+ Coverage   44.19%   44.31%   +0.12%     
==========================================
  Files        1348     1348              
  Lines       76184    76187       +3     
  Branches     6423     6543     +120     
==========================================
+ Hits        33670    33763      +93     
+ Misses      39244    39154      -90     
  Partials     3270     3270              

Flag	Coverage Δ
#go	47.75% <ø> (ø)
#storybook	4.38% <10.00%> (?)
#typescript	35.39% <10.00%> (+0.43%)
#unit	44.19% <0.00%> (-0.01%)

Impacted Files	Coverage Δ
shared/src/e2e/coverage.ts	0.00% <0.00%> (ø)
...d/src/components/activation/ActivationDropdown.tsx	48.57% <100.00%> (+48.57%)
browser/src/libs/options/ServerURLForm.tsx	95.55% <0.00%> (+2.22%)
shared/src/util/markdown.ts	80.00% <0.00%> (+3.33%)
...red/src/components/completion/CompletionWidget.tsx	25.00% <0.00%> (+5.00%)
web/src/nav/UserNavItem.tsx	68.57% <0.00%> (+11.42%)
web/src/components/tooltip/Tooltip.tsx	55.55% <0.00%> (+55.55%)
shared/src/notifications/NotificationItem.tsx	89.65% <0.00%> (+89.65%)
shared/src/components/activation/Activation.tsx	100.00% <0.00%> (+100.00%)

Created by: uwedeportivo

Dear all,

This is your release captain speaking.

Branch cut for the 3.16 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly. When in doubt, reach out!

Thank you

Created by: eseliger

2 things I noticed:

• Percy doesn't know it's only a subset of snapshots, so it complains about snapshots missing in the upload (the ones from e2e tests). Hence, I think once e2e runs all the snapshots will be marked as new and a diff check is lost
• Codecov cli logs the following:

==> Searching for coverage reports in:
--
  | + .
  | -> Found 3 reports
  | ==> Detecting git/mercurial file structure
  | ==> Reading reports
  | + ./shared/src/e2e/coverage.ts bytes=2180
  | + ./.storybook/coverage.test.ts bytes=704
  | + ./coverage/coverage-final.json bytes=1606867
  | ==> Appending adjustments

I don't know if it's an immediate problem, but they seemingly upload our two TS files as reports as well, probably because they are named coverage

Created by: felixfbecker

Ah, you're right :/ This sounds like a general problem if we have e2e tests screenshot and any other tests screenshot, and have e2e tests not run on every commit but the other tests. Any ideas for a solution?

Created by: felixfbecker

Configured with separate projects, I think this should solve it. https://docs.percy.io/docs/multiple-projects-per-repo

Created by: felixfbecker

I don't think they're harmful - Codecov will just discard them on the server. There's a flag to specify the file but I think I prefer the simpler command.

Created by: eseliger

I think we should estimate the cost of this as well - we generate a whole lot of snapshots for these tests, and percy charges per snapshot. We currently generate 41 of them - at 1 width for both Chrome and Firefox, which yields 82 snapshots per build in CI, in the past month, we had 2690 builds in BK, totalling to 220580 snapshots. At the first 10000 free, and $0.006/snapshot, that'll cost us an additional ((2690)*82-10000)*0.006=$1263.48

I think one way to reduce it is to disable Firefox in the percy project, which slashes it in half already

Created by: felixfbecker

That's a good point. I think we should also consider alternatives to Percy. For example, Chromatic has flat pricing tiers, is by the Storybook creators and has more features dedicated to Storybooks we could find valuable: It also hosts our storybook in the cloud and allows commenting and discussing it, which could be useful for our design process.

cc @rrhyne

Created by: eseliger

Where did you see that? I only find pricing tiers that are equally priced as Percy (per Snapshot and 0.006 ea)

Created by: felixfbecker

Ah I missed the "extra screenshot" price. It still seems though that with the same amount of snapshots, the Pro plan of Chromatic would be cheaper than Percy.

Created by: felixfbecker

@eseliger Percy has different plans - I think at that volume we wouldn't use the "Small" plan with 10k included anymore

• Small: 10k for $29/month plus $0.006 per additional screenshots
• Medium: 80k for $349/month plus $0.006 per additional screenshots
• Large: 200k for $849/month plus $0.006 per additional screenshots

Created by: eseliger

Filed https://gitlab.sgdev.org/root/sourcegraph/-/issues/10666