gitserver should auto-recover from corrupt repository clones

Created by: slimsag

An admin ran into an issue where a repository was corrupt and search was failing because of it:

1 error occurred: * diff search failed: failed to search commit diffs REDACTED@: git command [git log --no-prefix --max-count=31 --unified=0 --extended-regexp --regexp-ignore-case --all-match --author= -z --no-abbrev-commit --format=oneline --no-color --source --no-patch --no-merges -GREDACTED --regexp-ignore-case --pickaxe-regex --extended-regexp --] failed (output: ""): exit status 128 (stderr: "error: packfile .git/objects/pack/pack-REDACTED.pack does not match ... (truncated)")

This is the state the repository was in:

$ docker exec -it gitserver-7 sh -c 'cd /data/repos/REDACTED && time git log -- -n 1 HEAD'
error: packfile .git/objects/pack/pack-REDACTED.pack does not match index
error: packfile .git/objects/pack/pack-REDACTED.pack does not match index
error: packfile .git/objects/pack/pack-REDACTED.pack does not match index
error: packfile .git/objects/pack/pack-REDACTED.pack does not match index
error: packfile .git/objects/pack/pack-REDACTED.pack does not match index
error: packfile .git/objects/pack/pack-REDACTED.pack does not match index
fatal: bad object HEAD
Command exited with non-zero status 128
real    0m 0.00s
user    0m 0.00s
sys    0m 0.00s

Deleting the repository from disk manually was needed to recover from this. Gitserver should automatically handle this case somehow.

Reported by https://app.hubspot.com/contacts/2762526/company/407948923

cc @sourcegraph/core-services