An error occurred while fetching this tab.

vscode: add resolver-based graphql mocks for backcompat tests

Review changes
Download
Patches
Plain diff

Open vscode: add resolver-based graphql mocks for backcompat tests

tjk/vscode-new-gql-mock into main

Overview 4
Commits 3
Pipelines 0
Changes 19

Open vscode: add resolver-based graphql mocks for backcompat tests

Warren Giffordrequested to merge

tjk/vscode-new-gql-mock into main Jun 3, 2022

Overview 4
Commits 3
Pipelines 0
Changes 19

Created by: tjkandala

(review with whitespace hidden)

Problem:

We want to ensure that the VS Code extension delivers a quality experience to enterprise users. These users typically point the extension towards a Sourcegraph instance that is multiple versions behind the latest release. There are some bugs currently damaging the enterprise experience, and there may be new bugs introduced even as users' orgs upgrade as we iterate on feature code.

Now: Beatrix has done a great job manually testing and documenting bugs in the VS Code extension against each version, going back until the oldest supported version for core search features (v3.32). However, we would need to do more work to figure out why features are broken, while also running the risk of missing bugs. This GraphQL mocking technique exhaustively reveals bugs with error messages like:
- GraphQLError [Object]: Cannot query field "query" on type "SearchContext"
- GraphQLError [Object]: Unknown type "Event". Did you mean "Alert", "EventLog", or "Int"?
  - culprit of recent searches not working
Going forward:
- We, meaning the integrations team as well as other teams whose code we depend on, can move fast in feature code with the confidence that these guardrails bring. This system is resilient to changing operation names as well.
- Bumping the oldest supported version again. Speeds up the process of removing some feature detection/schema introspection code

Solution:

Add ability to run integration tests against older versions of the GraphQL API.

Define the oldest supported Sourcegraph version. For now I've chosen v3.22, since that's the oldest version for which search works. We download and check in the compressed GraphQL schema for that version by running yarn update-backcompat-schema in client/vscode (useful when bumping oldest supported version)
Add @graphql-tools/mock and @graphql-tools/schema to implement "resolver-based testing" vs "operation-based testing" (not sure if these are standard terms). Why couldn't we add backcompat tests with our existing setup?
1. We currently mock GraphQL requests by operation name, which could change over time. New queries could also work with old versions, which could cause false positive test failures.
2. It's a lot easier to test against the GraphQL schema directly as opposed to generating types for the old version as well as the new version, which would require downloading the whole TypeScript codebase and/or checking in the generated types.
3. We can now only specify mocks for fields we are interested in, since @graphql-tools/mock will return default values for scalar types.
Next Steps:
- In this PR I've only implemented the system for running VS Code extension integration tests against older Sourcegraph versions, but I haven't written all the mocks yet. I propose that we can fix a few bugs each iteration, after which we can declare the extension ready for enterprise users (on v3.22+) and write the necessary mocks (we'd only know which are necessary to write after bug fixes anyways). We can document the backcompat testing process in CONTRIBUTING.md then too.