Skip to content
Snippets Groups Projects

Take percy snapshots of storybooks

Merged Administrator requested to merge percy-storybooks into master

Created by: felixfbecker

Closes #9565 Closes #10631

Also simplifies our Storybook config (Storybook has new recommended config approaches since 5.3).

The coverage and uploading to Percy is separate, because Percy doesn't actually run our storybook in CI. It just gets all static HTML and uploads it to Percy to do the screenshotting in the cloud. We therefor track the coverage by loading each story in Puppeteer in a separate job.

This fixes one important bug that took a long time to find - our Puppeteer coverage reporter overwrote coverage files from previous tests. This must have been a problem for e2e coverage too.

Check out the Percy report in the GitHub status checks below, and the coverage report in the Codecov comment below.

Merge request reports

Approval is optional

Merged by avatar (Jul 8, 2025 7:40am UTC)

Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Created by: codecov[bot]

    Codecov Report

    Merging #10622 into master will increase coverage by 0.12%. The diff coverage is 10.00%.

    @@            Coverage Diff             @@
    ##           master   #10622      +/-   ##
    ==========================================
    + Coverage   44.19%   44.31%   +0.12%     
    ==========================================
      Files        1348     1348              
      Lines       76184    76187       +3     
      Branches     6423     6543     +120     
    ==========================================
    + Hits        33670    33763      +93     
    + Misses      39244    39154      -90     
      Partials     3270     3270              
    Flag Coverage Δ
    #go 47.75% <ø> (ø)
    #storybook 4.38% <10.00%> (?)
    #typescript 35.39% <10.00%> (+0.43%) :arrow_up:
    #unit 44.19% <0.00%> (-0.01%) :arrow_down:
    Impacted Files Coverage Δ
    shared/src/e2e/coverage.ts 0.00% <0.00%> (ø)
    ...d/src/components/activation/ActivationDropdown.tsx 48.57% <100.00%> (+48.57%) :arrow_up:
    browser/src/libs/options/ServerURLForm.tsx 95.55% <0.00%> (+2.22%) :arrow_up:
    shared/src/util/markdown.ts 80.00% <0.00%> (+3.33%) :arrow_up:
    ...red/src/components/completion/CompletionWidget.tsx 25.00% <0.00%> (+5.00%) :arrow_up:
    web/src/nav/UserNavItem.tsx 68.57% <0.00%> (+11.42%) :arrow_up:
    web/src/components/tooltip/Tooltip.tsx 55.55% <0.00%> (+55.55%) :arrow_up:
    shared/src/notifications/NotificationItem.tsx 89.65% <0.00%> (+89.65%) :arrow_up:
    shared/src/components/activation/Activation.tsx 100.00% <0.00%> (+100.00%) :arrow_up:
  • Created by: uwedeportivo

    Dear all,

    This is your release captain speaking. :steam_locomotive::steam_locomotive::steam_locomotive:

    Branch cut for the 3.16 release is scheduled for tomorrow.

    Is this issue / PR going to make it in time? Please change the milestone accordingly. When in doubt, reach out!

    Thank you

  • Created by: eseliger

    2 things I noticed:

    • Percy doesn't know it's only a subset of snapshots, so it complains about snapshots missing in the upload (the ones from e2e tests). Hence, I think once e2e runs all the snapshots will be marked as new and a diff check is lost
    • Codecov cli logs the following:
    ==> Searching for coverage reports in:
    --
      | + .
      | -> Found 3 reports
      | ==> Detecting git/mercurial file structure
      | ==> Reading reports
      | + ./shared/src/e2e/coverage.ts bytes=2180
      | + ./.storybook/coverage.test.ts bytes=704
      | + ./coverage/coverage-final.json bytes=1606867
      | ==> Appending adjustments

    I don't know if it's an immediate problem, but they seemingly upload our two TS files as reports as well, probably because they are named coverage :grin:

  • Created by: felixfbecker

    Ah, you're right :/ This sounds like a general problem if we have e2e tests screenshot and any other tests screenshot, and have e2e tests not run on every commit but the other tests. Any ideas for a solution?

  • Created by: felixfbecker

    Configured with separate projects, I think this should solve it. https://docs.percy.io/docs/multiple-projects-per-repo

  • Created by: felixfbecker

    I don't think they're harmful - Codecov will just discard them on the server. There's a flag to specify the file but I think I prefer the simpler command.

  • Created by: eseliger

    I think we should estimate the cost of this as well - we generate a whole lot of snapshots for these tests, and percy charges per snapshot. We currently generate 41 of them - at 1 width for both Chrome and Firefox, which yields 82 snapshots per build in CI, in the past month, we had 2690 builds in BK, totalling to 220580 snapshots. At the first 10000 free, and $0.006/snapshot, that'll cost us an additional ((2690)*82-10000)*0.006=$1263.48

    I think one way to reduce it is to disable Firefox in the percy project, which slashes it in half already

  • Created by: felixfbecker

    That's a good point. I think we should also consider alternatives to Percy. For example, Chromatic has flat pricing tiers, is by the Storybook creators and has more features dedicated to Storybooks we could find valuable: It also hosts our storybook in the cloud and allows commenting and discussing it, which could be useful for our design process.

    cc @rrhyne

  • Created by: eseliger

    Where did you see that? I only find pricing tiers that are equally priced as Percy (per Snapshot and 0.006 ea)

  • Created by: felixfbecker

    Ah I missed the "extra screenshot" price. It still seems though that with the same amount of snapshots, the Pro plan of Chromatic would be cheaper than Percy.

  • Created by: felixfbecker

    @eseliger Percy has different plans - I think at that volume we wouldn't use the "Small" plan with 10k included anymore

    • Small: 10k for $29/month plus $0.006 per additional screenshots
    • Medium: 80k for $349/month plus $0.006 per additional screenshots
    • Large: 200k for $849/month plus $0.006 per additional screenshots
Please register or sign in to reply
Loading