Write OOB migration to move repos from md5 hashing to rendezvous hashing
View options
- Truncate descriptions
Created by: ryanslade
For context, see this doc
As we switch from our current md5 hashing to rendezvous hashing, repos will need to be moved from one instance of gitserver to another.
We will need to write an OOB migrator.
We will iterate over all repos in repo name order. For each repo we need to do the following:
- Find it's new home based on rendezvous hashing
- Clone the repo to it's new location
- Record that the repo has been moved. This can be done by storing the name in the metadata column for the migration
- Remove the repo from it's old location
Once a repo has been moved, we want all new reads to use rendezvous hashing.
To do this we need to poll the migration to find out the latest repo that has been moved. If the repo we want to read falls before that repo in order, use the new hash, otherwise, use the old hash. The polling can be as low as once per second, what we want to avoid is hitting the database on every hash request.
Things to consider:
- Some repos may fail to clone. We should make a best effort, maybe try 3 times before giving up and marking it as moved so that we can continue the migration.
- The
Up()
method on theMigrator
interface is expected to migrate a batch of repos. We may want to grab a few hundred repos at a time. We can perform a blocking clone so that we don't exceed limits set byGitMaxConcurrentClones
- Not all repos are currently cloned, we probabaly only want to consider repos marked as
cloned
ingitserver_repos
. It's wasteful to move uncloned repos.
Success criteria:
- We should be able to disable the migration via an envvar so that we can deploy to dogfood and run a full migration there before we try on .com or customer instances
- We should be able to make forward progress with a best effort to move all repos
- We should be able to quickly stop the migration if we spot errors
- The migration should not adversely affect performance


- Show labels
- Show closed items
Link items together to show that they're related.