Skip to content

Code coverage: Improve and split Storybook coverage from test coverage in Codecov

Created by: umpox

Description

We currently generate code coverage for both our tests (unit/integration/e2e) and our Storybook stories (visual regression testing). This has some problems:

  • Difficult to determine how much of our code is tested vs how much is covered by Storybook
  • Can give us false confidence in the amount of testing we have in our codebase.
    • Storybook tests are akin to snapshot tests - which we want to move away from relying on, as they no not necessarily test functionality, and can be too easy to skim over and update
  • The Storybook step in particular is very slow (currently ~26 minutes average). Relevant Slack thread
  • Storybook step isn't fully accurate right now. It doesn't account for stories where we have disabled visual snapshotting, this was a recent change to reduce cost.

We discussed the value of this setup with the rest of frontend devs and found the following:

  • Coverage is useful (obviously)
  • Storybook coverage is still useful to determine how much of our codebase is covered by visual tests, and also to capture some underused components (e.g. a Page wrapper).
    • Note: The latter point here would be solved by mocking towards a complete user journey test approach (minimal mocking, render as much as possible in test)

Solution

We should aim to do the following:

  • Split the Storybook coverage into its own category, so we can easily split coverage between actual tests and the stories. This might be possible through Codecov flags
  • Investigate and speed up the Storybook coverage step in the sourcegraph-async pipeline.
    • Note: Make additional ticket for this if required
  • Improve accuracy of Storybook coverage by only including components that have opted-in to visual snapshots
    • Note: Make additional ticket for this if required