Implement a mechanism to report CI tooling failures on Sentry

Created by: jhchabran

In most cases, when a command fails on Sentry, it'll lead to a failed build, which is the expected behaviour. But in some rarer cases, it's because the tooling failed in some way.

That's particularly troublesome because it can easily be missed unless someone recognised and reported the failure. Independently on how we decide to evolve the CI, this will come handy as there always be a bit of shell script involved and this will make sure we're noticed.

Failures that we have witnessed so far:

• Pipeline generation failures
• Build uploads failures
• sg compilation errors that prevented build uploads

Goals

• Keep it as small as possible.
• Must be standalone and not environment dependent (limiting possible source of failures, which is mandatory for a reporting tool

Non-Goals

• Building something custom.

Solution discovery

A potential solution would be to have a command that we can use to wrap commands that should never fail such as xyz || sentry-report-failure.

Sentry cli seems to fill the requirements already: https://docs.sentry.io/product/cli/send-event/#events-with-parameters

I would have to be available by default to the agents, so we aren't dependent on another compilation step that could fail to make the reporting command available.

Discussion

• @bobheadxi this strongly relates to RFC 496