search: prevent issuing more than 200+ unindexed searches for a single query

Created by: slimsag

Today, if a search query is issued and no repositories are indexed all repositories will be searched using git grep via a request to searcher.

Historically, this has been desirable for a few reasons:

1. sourcegraph/server used to not perform any indexed search, and as such it was entirely based on live searching.
2. We advised customers running Kubernetes clusters to deploy a large worker pool of searchers capable of handling a large amount of requests (partially because we in the early days did not have any indexing at all).

Over time, we drifted away from the two above cases: sourcegraph/server got indexing and everyone turned it on by default; and customers (and us) began to realize unindexed searching was uncommon and dedicating a ton of resources to it was not wise.

Nowadays, most Sourcegraph deployments only have a few unindexed searchers - primarily for handling the case in which someone wants to search on unindexed non-HEAD branches.

The fact that we can still fire off several thousand unindexed search requests to searcher is bad and silly for multiple reasons:

• In early 2019 (link), this caused network outages on the host machines running the searcher containers because we would literally flood the network with requests to searcher. At the time, we solved this by limiting the number of requests and this was sufficient to solve the network DOS we were causing ourselves.
• Large amounts of unindexed symbol searches have been a major cause for other resource contention in the past (primarily in sourcegraph/server deployments).
• When we introduced the new tab-based search UX, it directed users to run e.g. type:diff searches globally that would result in thousands of unindexed diff searches, which caused major instability issues and a poor UX for many users until we restricted it to search only 50 repositories at a time.
• ("the issue") Today, on a very large $CUSTOMER replica with 400k+ repositories we find a similar issue: 380k repositories are indexed, but due to other issues 20k are not. This causes an otherwise very fast indexed search that only takes a few seconds to spend a very long time trying to complete 20k unindexed search requests to searcher - often just resulting in the search request timing out entirely.

The fact that we allow this at all is silly:

• No user wants to wait a long time just to get a timeout
• No admin wants their users to be able to run searches that can take down / harm Sourcegraph.

We almost solely had this behavior for historical reasons, and have been slowly walking it back.

This PR adds one final strict guarantee to fix "the issue" above: A single search query cannot result in more than 200 repo@revs being searched by our unindexed search codepath:

If your search query would result in more than 200+ unindexed repo@revs being searched, we search the first 200 and report the remaining repositories as missing in the web UI. In this case, you can hover over the list to see which were omitted. This is the same way we report repositories as unsearchable in a few other scenarios (e.g. if you run an index:only search and they are not in the index, or if there is an issue with the repository).

This merely mitigates the underlying issue with a good-enough (albeit not perfect) UX, until us or a site admin can fix the real underlying issue: the repositories aren't indexed.

Helps https://sourcegraph.slack.com/archives/CTQ5LTF8D / https://app.hubspot.com/contacts/2762526/company/557692805/