endpoint: add rendezvous hashing support
Created by: tsenart
This commit adds rendezvous hashing support to the endpoint package with the aim to make it the default in a follow up PR. It's currently protected by an environment variable which I plan to set on dogfood for testing.
Additionally, it paves the way for increasing the replication factor by
adding a GetN
method to map, which I used to replace the previous
exclude
paramater to Get
(used only by searcher). Actually raising
replication factor for Zoekt will come later.
Here is the output of the tests, which shows much better distribution, and illustrates the data movement needed when changing hash function and cluster sizes.
I think it's OK for us to shepherd this to prod next week and treat it like and index version bump, and to write some release notes for customers to expect some downtime in the next upgrade.