Add experimental paginated search API

Created by: slimsag

This PR adds an experimental paginated search API. There are a few caveats:

1. There are some known limitations (see docs added).
2. The test coverage is not sufficient at all currently (but, it is very clearly marked as experimental).
3. This is just an API, I would still like to add a src counterpart change. All existing clients, including the web UI, continue to use the standard non-paginated API and may continue to do so for the foreseeable future. This is currently just useful for programatic consumption of results.

That said, I would like to merge this as-is anyway because:

1. This is a very low-risk change as it is a completely separate codepath from the rest of our search, and is clearly marked as experimental at all relevant locations.
2. This is at a good "stopping point" where it can be merged, which allows me to avoid this PR becoming larger and more not easily reviewed. After merging, I can send more isolated changes to add more extensive testing and lift known limitations in isolated PRs that are more easily reviewed.
3. The next major milestone here will involve testing this at larger scales on e.g. k8s.sgdev.org and sourcegraph.com.

Current limitation

The primary limitation for now is that this only operates on type:file (text) search results. If you do not specify that (or specify any other type: in your query) you will receive an error. This should be an easy limitation to lift, and the implementation is generalizable to all result types -- I just haven't done so yet as there is one last detail here to sort out (what order guarantee we make about indexed and unindexed results being intermixed). Testing at larger scales will help me to make this decision.

Reviewing

I suggest first reading the architecture document added in this PR which outlines how this works at a high-level, links to the API documentation which is quite extensive and covers many edge cases, and then looking at the implementation once you've read those aspects.

Trying it out

To try this out, you can run this branch locally and then run a query like this in the API console:

{
  search(query: "type:file Forbidden", first: 5, cursor: null) {
    results {
      matchCount
      results {
        __typename
        ... on FileMatch {
          repository {
            name
          }
        }
      }
      pageInfo {
        endCursor
        hasNextPage
      }
    }
  }
}

You can then pass the returned pageInfo.endCursor base64 cursor ID into a new GraphQL query to get more results:

{
  search(query: "type:file Forbidden", first: 5, cursor: "U2VhcmNoQ3Vyc29yOnsiUmVwb3NpdG9yeU9mZnNldCI6MiwiUmVzdWx0T2Zmc2V0IjoyLCJGaW5pc2hlZCI6ZmFsc2UsIlVzZXJJRCI6MH0=") {
    results {
      matchCount
      results {
        __typename
        ... on FileMatch {
          repository {
            name
          }
        }
      }
      pageInfo {
        endCursor
        hasNextPage
      }
    }
  }
}

And repeat this process until pageInfo.hasNextPage is true to get all results. See the added documentation for a more in-depth explanation.