Take percy snapshots of storybooks
Created by: felixfbecker
Closes #9565 Closes #10631
Also simplifies our Storybook config (Storybook has new recommended config approaches since 5.3).
The coverage and uploading to Percy is separate, because Percy doesn't actually run our storybook in CI. It just gets all static HTML and uploads it to Percy to do the screenshotting in the cloud. We therefor track the coverage by loading each story in Puppeteer in a separate job.
This fixes one important bug that took a long time to find - our Puppeteer coverage reporter overwrote coverage files from previous tests. This must have been a problem for e2e coverage too.
Check out the Percy report in the GitHub status checks below, and the coverage report in the Codecov comment below.
Merge request reports
Activity
Created by: codecov[bot]
Codecov Report
Merging #10622 into master will increase coverage by
0.12%
. The diff coverage is10.00%
.@@ Coverage Diff @@ ## master #10622 +/- ## ========================================== + Coverage 44.19% 44.31% +0.12% ========================================== Files 1348 1348 Lines 76184 76187 +3 Branches 6423 6543 +120 ========================================== + Hits 33670 33763 +93 + Misses 39244 39154 -90 Partials 3270 3270
Flag Coverage Δ #go 47.75% <ø> (ø)
#storybook 4.38% <10.00%> (?)
#typescript 35.39% <10.00%> (+0.43%)
#unit 44.19% <0.00%> (-0.01%)
Impacted Files Coverage Δ shared/src/e2e/coverage.ts 0.00% <0.00%> (ø)
...d/src/components/activation/ActivationDropdown.tsx 48.57% <100.00%> (+48.57%)
browser/src/libs/options/ServerURLForm.tsx 95.55% <0.00%> (+2.22%)
shared/src/util/markdown.ts 80.00% <0.00%> (+3.33%)
...red/src/components/completion/CompletionWidget.tsx 25.00% <0.00%> (+5.00%)
web/src/nav/UserNavItem.tsx 68.57% <0.00%> (+11.42%)
web/src/components/tooltip/Tooltip.tsx 55.55% <0.00%> (+55.55%)
shared/src/notifications/NotificationItem.tsx 89.65% <0.00%> (+89.65%)
shared/src/components/activation/Activation.tsx 100.00% <0.00%> (+100.00%)
Created by: eseliger
2 things I noticed:
- Percy doesn't know it's only a subset of snapshots, so it complains about snapshots missing in the upload (the ones from e2e tests). Hence, I think once e2e runs all the snapshots will be marked as new and a diff check is lost
- Codecov cli logs the following:
==> Searching for coverage reports in: -- | + . | -> Found 3 reports | ==> Detecting git/mercurial file structure | ==> Reading reports | + ./shared/src/e2e/coverage.ts bytes=2180 | + ./.storybook/coverage.test.ts bytes=704 | + ./coverage/coverage-final.json bytes=1606867 | ==> Appending adjustments
I don't know if it's an immediate problem, but they seemingly upload our two TS files as reports as well, probably because they are named
coverage
Created by: felixfbecker
Configured with separate projects, I think this should solve it. https://docs.percy.io/docs/multiple-projects-per-repo
Created by: eseliger
I think we should estimate the cost of this as well - we generate a whole lot of snapshots for these tests, and percy charges per snapshot. We currently generate 41 of them - at 1 width for both Chrome and Firefox, which yields 82 snapshots per build in CI, in the past month, we had 2690 builds in BK, totalling to 220580 snapshots. At the first 10000 free, and $0.006/snapshot, that'll cost us an additional ((2690)*82-10000)*0.006=$1263.48
I think one way to reduce it is to disable Firefox in the percy project, which slashes it in half already
Created by: felixfbecker
That's a good point. I think we should also consider alternatives to Percy. For example, Chromatic has flat pricing tiers, is by the Storybook creators and has more features dedicated to Storybooks we could find valuable: It also hosts our storybook in the cloud and allows commenting and discussing it, which could be useful for our design process.
cc @rrhyne
Created by: felixfbecker
@eseliger Percy has different plans - I think at that volume we wouldn't use the "Small" plan with 10k included anymore
- Small: 10k for $29/month plus $0.006 per additional screenshots
- Medium: 80k for $349/month plus $0.006 per additional screenshots
- Large: 200k for $849/month plus $0.006 per additional screenshots
Created by: eseliger
Filed https://gitlab.sgdev.org/root/sourcegraph/-/issues/10666