buildchecker: pipeline failure detection and branch locking
Created by: bobheadxi
This will run as an automated cron every ~15mins, and it will do the following:
- get a page of recent builds and check if:
- the most recent build is failed
- that failure is consecutive
- if no issues detected, remove branch push restrictions
- if restrictions were removed, post in slack
- if an issue is detected:
- try and find github handles for the relevant authors of the commits
- lock main branch: this is done by restricting push to these build authors + the devx team
- post in slack if this is a new restriction (existing restrictions won't be overwritten)
I've tried to include some extensive testing because this is definitely not something we want misbehaving
Close https://github.com/sourcegraph/sourcegraph/issues/25482 and https://github.com/sourcegraph/sourcegraph/issues/28834 and https://github.com/sourcegraph/sourcegraph/issues/29005
Notes:
- A lot of the diff is just generated API interaction recordings
-
A major caveat is that the API does not currently allow us to set "require pull request" (!!!!) https://github.com/sourcegraph/sourcegraph/pull/28759/files#r767060192 . Is this a blocker?This is fixed by using the GraphQL API for this partworkaround no longer needed due to https://github.com/sourcegraph/sourcegraph/issues/29005 - Currently does not automate incident creation, but does encourage it in the slack message
- Currently does not explicitly tag users, since that is complicated and might be better served by https://github.com/sourcegraph/sourcegraph/issues/25454. A slack message is posted though