Skip to content

endpoint: add rendezvous hashing support

Warren Gifford requested to merge endpoints-better-consistent-hashing into main

Created by: tsenart

This commit adds rendezvous hashing support to the endpoint package with the aim to make it the default in a follow up PR. It's currently protected by an environment variable which I plan to set on dogfood for testing.

Additionally, it paves the way for increasing the replication factor by adding a GetN method to map, which I used to replace the previous exclude paramater to Get (used only by searcher). Actually raising replication factor for Zoekt will come later.

Here is the output of the tests, which shows much better distribution, and illustrates the data movement needed when changing hash function and cluster sizes.

I think it's OK for us to shepherd this to prod next week and treat it like and index version bump, and to write some release notes for customers to expect some downtime in the next upgrade.

Merge request reports

Loading