Skip to content

dev/ci: conditional builds for executor

Warren Gifford requested to merge executor-patch-notest into main

Created by: bobheadxi

Introduce a pair of CI scripts and a new standard, though all this only applies for the Executor builds at the moment:

  • .../${build_dir}/hash.sh: output some identifier(s) where, if different across revisions, indicates a significant change has been made that deserves a rebuild
  • enterprise/dev/ci/scripts/compare-hash.sh: compares the output of a given hash.sh between the current revision and the previous. If nothing has changed, exits with a special code, 222.
  • enterprise/dev/ci/scripts/check-dependency-soft-fail.sh: if the outcome of the given step is soft_fail, exit with a special code, 222.

If hash checking is not disabled, pipeline is generated such that exit code 222 is treated as a soft failure that allows the subsequent steps to continue. The publish step checks if the build step exited with 222 and either proceeds with a publish, or soft fails as well.

Why not conditionals? It doesn't seem like Buildkite conditionals allow arbitrary scripts, nor do the built-in conditionals allow us to see the outcome of a previous step. GitHub Actions seem to win here 🥲 Spent a while trawling through docs for alternatives, and I think this is the best approach.

Also introduces MessageFlags, which allow us to specify [flag]-style message indicators to tweak pipeline behaviour. Keeping the complexity of this down will probably be tricky but I'd rather this than 30 RunTypes for every pipeline imaginable.

  • Migrate enabling profiling to [buildkite-enable-profiling] messages instead of a branch
  • Add [skip-hash-compare] to trigger a build that doesn't check for the hash

Lastly, this also adds a RunType for the executor-patch-notest branch, which mimics docker-images-patch-notest, but for the executor pipeline. This PR is from that branch at the moment, so please don't push to it yet 🙃

See examples

I'm hoping to potentially generalize the hash.sh pattern for the Docker image builds in the future as well. This could mean we can potentially drop arbitrary schedules from Renovate in favour of services rolling out when they actually change. Could save on costs too.

Helps close https://github.com/sourcegraph/sourcegraph/issues/25487

Merge request reports