Skip to content

gigarepo corrupted on k8s.sgdev.org - gitserver isn't automatically resolving the issue

Created by: ggilmore

The gigarepo on k8s.sgdev.org seems like it's corrupted, but our automatic re-clone logic in gitserver isn't detecting this case.

For example: https://sourcegraph.grafana.net/explore?orgId=1&left=%7B%22datasource%22:%22grafanacloud-sourcegraph-traces%22,%22queries%22:%5B%7B%22query%22:%222f546ac40c74c0b3%22,%22queryType%22:%22traceId%22,%22refId%22:%22A%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D

These commands show off the corrupted clone on the gitserver instance:

# on gitserver-3
bash-5.1$ pwd
/data/repos/gigarepo
bash-5.1$ git log
fatal: bad object HEAD
bash-5.1$ git rev-parse HEAD
184bcb864ec879803591fda4d5773d2aecf1fd38
bash-5.1$ git show 184bcb864ec879803591fda4d5773d2aecf1fd38
fatal: bad object 184bcb864ec879803591fda4d5773d2aecf1fd38
bash-5.1$ git ls-tree --long --full-name -z 184bcb864ec879803591fda4d5773d2aecf1fd38 -- linux/drivers/net/wireless/ath/wcn36xx/testmode_i.h
fatal: not a tree object
bash-5.1$

These commands show that the original gigarepo on git-combine isn't corrupted:

# original repository on git-combine
/data/repos/gigarepo/.git # git log | head
commit 37ef72f86a22cce44d56884d4949edaebf66ec44
Author: First Contributions Bot <[email protected]>
Date:   Mon Aug 15 15:33:55 2022 +0200

    first-contributions: Merge pull request #52396 from RossDScott/first-contributions

    Commit: 706ac981eb82d2a67d20bcdb3d18b6c0e55d2ac3

commit 4849a4793ffcf8a0f71f2b16981eae6222247f61
Author: Daniel Lockyer <[email protected]>
/data/repos/gigarepo/.git # git show 37ef72f86a22cce44d56884d4949edaebf66ec44 | head
commit 37ef72f86a22cce44d56884d4949edaebf66ec44
Author: First Contributions Bot <[email protected]>
Date:   Mon Aug 15 15:33:55 2022 +0200

    first-contributions: Merge pull request #52396 from RossDScott/first-contributions

    Commit: 706ac981eb82d2a67d20bcdb3d18b6c0e55d2ac3

diff --git a/first-contributions/Contributors.md b/first-contributions/Contributors.md
index 42e2a1e3437ba..056ea087ffde1 100644
/data/repos/gigarepo/.git # git rev-parse HEAD | head
37ef72f86a22cce44d56884d4949edaebf66ec44
/data/repos/gigarepo/.git #  git ls-tree --long --full-name -z 37ef72f86a22cce44d56884d4949edaebf66ec44 --
 linux/drivers/net/wireless/ath/wcn36xx/testmode_i.h
100644 blob 8a1477ffd5a0fc3a91f53e11bf9c71733f338f8a    1095	linux/drivers/net/wireless/ath/wcn36xx/testmode_i.h/data/repos/gigarepo/.git #

I made a video of the bad UX that I'm seeing when I run this search:

https://user-images.githubusercontent.com/9022011/184645825-1189897e-e8b6-4708-bd15-bd943437b6c2.mov

I'm going to just force a reclone of the gigarepo to unblock myself, but @sourcegraph/repo-management should probably consider if there is any detection logic that we could add to gitserver to help detect this failure mode.

/cc @jplahn @ryphil