Skip to content

Improve search regression tests

Administrator requested to merge bl/regression-testing into master

Created by: beyang

Move the regression test suite (for automated release testing) from web/src/e2e to web/src/regression. This decouples the regression testing suite (which is intended to be run manually or by a special service against different Sourcegraph instances) from the e2e testing suite (which is intended to be run in CI against a fresh Sourcegraph instance).

Also implements the following:

  • Adds a config.ts module that encapsulates logic for reading in test config. Both the e2e and regression tests need to be fed config fields like Sourcegraph username, auth token, and GitHub token. This module defines a config schema (the Config interface), which is shared between the e2e and regression tests. The regression tests use getConfig to declare which config fields they depend on and read these from the test environment (either env vars or a config file). The e2e tests have been updated to use getConfig to read in some config, but I haven't yet gotten around to migrating all the e2e env vars to use this function. I plan to do so in a future PR.
  • Adds modules api.ts and apiDriver.ts for interacting with the Sourcegraph API. These are useful for ensuring certain preconditions hold on the Sourcegraph instance (e.g., an external service with a particular name exists or must be added before the tests run). Such preconditions were previously enforced by having the Puppeteer driver execute UI actions, but this was slow to run and tedious to write. Using the API driver, rather than the UI driver, to ensure these preconditions hold decreases the test runtime from 86s to 13.5s.
  • Fixed broken and incomplete tests added in https://github.com/sourcegraph/sourcegraph/pull/5302, so the regression test suite actually passes again.

To run the regression test suite, do the following:

  • Spin up a Sourcegraph instance (recommended you do this against a dev version, since the regression tests will modify the instance to ensure certain repositories exist)
  • Create a sudo-level access token on the Sourcegraph instance
  • cd web and run SOURCEGRAPH_USERNAME=$USER SOURCEGRAPH_SUDO_TOKEN=$SUDO_TOKEN GITHUB_TOKEN=$(cat ../../dev-private/enterprise/dev/external-services-config.json | grep sqs-test | sed "s/.*: \"\(.*\)\".*/\1/") SOURCEGRAPH_BASE_URL=http://localhost:3080 yarn run test-regression (replacing $SUDO_TOKEN with the token you just created and $USER with the Sourcegraph username you want to run the test as.

@chrismwendt it would be great if you could review this since you authored the e2e test suite. @felixfbecker would be great to get your feedback on TS style.

After this is merged, I plan to implement more regression tests in the web/regression directory.

Merge request reports