Skip to content

search: Introduce Indexers for sharding indexed search

Warren Gifford requested to merge core/zoekt-horizontal-sharding-assign into master

Created by: keegancsmith

Indexers will contain functionality related to splitting up searching and indexing across multiple shards. This commit introduces the use of our endpoint pkg to shard repositories amongst indexed search pods.

The hostname an indexed search pod advertises to us will likely be different to the address we use to contact it. As such we have logic to match it up with an endpoint address.

The endpoint package also allows us to use service discovery in the future (like we do with searcher and k8s). However, we won't use it yes since it will require more extensive testing first.

This functionality is currently all feature flagged behind the environment variable INDEXED_SEARCH_SERVERS. This will remain empty until we are happy to enabled horizontal scaling by default.

Note: This only implements assignment of repos to a replica at indexing time as described in v0 of the horizontal search RFC. Another PR will implement the logic which splits up the search amongst multiple replicas.

Test Plan: Tested that this code path is ignored in dev mode. Also have some dev mode magic to test horizontal search which will appear in its own PR.

Part of https://github.com/sourcegraph/sourcegraph/issues/5725

Merge request reports

Loading