Attempt to fix a corrupted repo instead of recloning

Created by: indradhanush

Currently, if a repo is found to be corrupt we attempt to reclone it endlessly. There is some initial finding which seems to indicate that sg maintenance might be conflicting with git's default garbage collection leading to the repo corruption.

Instead of attempting to reclone, we should attempt to fix the corrupted refs instead.

One of our customers have used this script to fix the repo and were kind to share it with us:

#!/usr/bin/env bash
function clean_invalid_refs() {
    INVALID_SHAS=`git for-each-ref --format="%(objectname)" | sort | uniq | git cat-file --batch-check | grep missing | cut -d' ' -f 1`;
    if [ -z "${INVALID_SHAS}" ]; then
        echo "Nothing to cleanup" > /dev/stderr
        return
    fi
    
    REGEX=$(echo "$INVALID_SHAS" | paste -sd '|')
    REFS=`git for-each-ref --format="%(refname) %(objectname)" | grep -E $REGEX | cut -d' ' -f 1`
    for ref in $REFS; do
        echo "deleting ${ref}"
        git update-ref -d ${ref}
    done
}

clean_invalid_refs

git fetch "$@"

This should serve as a great starting point for this effort.

Important

• Create a metric to track each time a corrupted repo is "fixed"
• Optionally, create an alert (owned by @sourcegraph/repo-management) if the number of such attempted fixes is higher than a given threshold (tbd) within a particular time period (also tbd)