incremental search proposal / technical research

Created by: slimsag

Explanation

See this public google doc

State of this PR

• Implements a repository-level pagination API, both in the backend and GraphQL layer.
• I spent maybe ~8hr total on this + the doc above, so it proves my idea would be a quick way for us to support this from a technical standpoint.
• The GraphQL layer is just for seeing what it would look like if we did expose repository-level pagination via GraphQL. It would NOT be actually committed if we went forward with my proposal / PR here. Instead, it would be replaced with a cursor-based result-level pagination API which has multiple benefits over exposing the repository-level API as I discovered and detailed in my proposal google doc.
• The backend implementation of this would be used if we moved forward with this.

Not done in this PR currently, but would be done if we move forward with this approach:

• Building the cursor-based result-level pagination GraphQL API which has multiple benefits over the repository-level pagination GraphQL API exposed in this PR, it would improve performance considerably.
• The webapp / TypeScript implementation of this, depends on the above.
• Support for non-text-matches, i.e. queries that include symbol search results or file path search results. Have to add type:file to your query to limit results to just text matches (which are the most complex in terms of implementation) for it to work right now.
• Proper tuning of the concurrency division, with cursor-based result-level API we could do more concurrent work to get results even quicker. e.g. right now we do totalRepos/8 but we could divide that further into concurrent work to potentially return to the client much sooner.
• Proper testing of this implementation.

Trying this out

To try this out, you can run the dev server normally then use the API console like this