Skip to content
Snippets Groups Projects
Closed Write OOB migration to move repos from md5 hashing to rendezvous hashing
  • View options
  • Write OOB migration to move repos from md5 hashing to rendezvous hashing

  • View options
  • Closed Issue created by Warren Gifford

    Created by: ryanslade

    For context, see this doc

    As we switch from our current md5 hashing to rendezvous hashing, repos will need to be moved from one instance of gitserver to another.

    We will need to write an OOB migrator.

    We will iterate over all repos in repo name order. For each repo we need to do the following:

    1. Find it's new home based on rendezvous hashing
    2. Clone the repo to it's new location
    3. Record that the repo has been moved. This can be done by storing the name in the metadata column for the migration
    4. Remove the repo from it's old location

    Once a repo has been moved, we want all new reads to use rendezvous hashing.

    To do this we need to poll the migration to find out the latest repo that has been moved. If the repo we want to read falls before that repo in order, use the new hash, otherwise, use the old hash. The polling can be as low as once per second, what we want to avoid is hitting the database on every hash request.

    Things to consider:

    • Some repos may fail to clone. We should make a best effort, maybe try 3 times before giving up and marking it as moved so that we can continue the migration.
    • The Up() method on the Migrator interface is expected to migrate a batch of repos. We may want to grab a few hundred repos at a time. We can perform a blocking clone so that we don't exceed limits set by GitMaxConcurrentClones
    • Not all repos are currently cloned, we probabaly only want to consider repos marked as cloned in gitserver_repos. It's wasteful to move uncloned repos.

    Success criteria:

    • We should be able to disable the migration via an envvar so that we can deploy to dogfood and run a full migration there before we try on .com or customer instances
    • We should be able to make forward progress with a best effort to move all repos
    • We should be able to quickly stop the migration if we spot errors
    • The migration should not adversely affect performance

    Linked items 0

  • Link items together to show that they're related.

    Activity

    • All activity
    • Comments only
    • History only
    • Newest first
    • Oldest first
    Loading Loading Loading Loading Loading Loading Loading Loading Loading Loading