search: Introduce HorizontalSearcher for indexed search
Created by: keegancsmith
This is similiar to shardedSearcher in the zoekt codebase. AggregateSearcher will aggregate search results over all indexed search pods stored in the endpoint map. It will open and close clients as addresses come and go in the endpoint map. It takes care to have a fast path for the common case (no change in the endpoint map).
This implementation is as described in the v0 of the RFC 30: Zoekt Horizontal Scaling. AggregateSearcher is a zoekt.Searcher such that the rest of the Sourcegraph codebase isn't aware there are multiple indexed search pods.
The main shortcoming of this implementation is around reliability and correctness. If any endpoint is down, then the whole of indexed search will be down. It can return duplicates in the case of the set of indexed search pods changing. For small number of replicas (2 or 3) that very rarely change, this isn't much worse than when we have a single indexed search pod. I expect this to be the common case, as such this implementation may be good enough for most customers.
Note: This is still disabled by default. A customer will have to explicitly configure INDEXED_SEARCH_SERVERS.
Test plan: Unit tests and manual testing in dev mode (check if results are returned from all replicas).
Part of https://github.com/sourcegraph/sourcegraph/issues/5725