Experimental search pagination / incremental search API

Created by: slimsag

Today, it is only possible to retrieve whether or not your search query has more results via e.g. limitHit. You must then retry your query and request a higher number of results via the count: search query filter.

It would be nice if we had proper pagination via an offset + count.

Why

Running Sourcegraph with thousands of repositories and executing a heavy/large query on all of those is expensive, and can take several minutes depending on the query (for example, symbol searches). This results in timeouts due to (1) a 1-minute-max hard-coded search timeout and (2) it is desirable to have the HTTP proxy that in front of Sourcegraph have sane timeouts like e.g. 2 minutes. Thus, this is beneficial because:

1. Allows scripts to workaround search timeouts.
2. Allows scripts to consume search results incrementally.
3. For users who care, allows searching for results in a specified order via adding e.g. offset:0 to your query.

Implementation

The implementation would be something like passing a pagination parameter down through the search pipeline which controls whether or not we construct search results in a stable order or not. By doing this, we can make smart choices to enable both paginated search to be fast AND still have fast time-to-result unstable ordering for the webapp UI.

Reported by https://app.hubspot.com/contacts/2762526/company/407948923