dev/ci: introduce and migrate to bk.AnnotatedCmd

Created by: bobheadxi

bk.AnnotatedCmd is the generator-based pull-oriented annotations API to replace sporadic calls to annotate.sh and buildkite-angent annotate, splitting out annotation creation concerns out of check scripts and into the pipeline generator in a style similar to how ArtifactPaths works, but cooler and with way more bad bash from an engineer desperately copy-pasta-ing from Stackoverflow late at night (myself).

This achieves all of the goals laid out by @jhchabran in https://github.com/sourcegraph/sourcegraph/issues/26071#issuecomment-952680939:

• It should be simple enough that you can add annotations from any shell script → see usage below
• Do not break the scripts if not run on the CI → biggest win here - we completely detach annotation commands from check scripts, and make annotations meaningful locally as well since you can find them in ./annotations. Potential sg check integration maybe? 👀
• Define what is a good (and bad) annotation and document it. → with bk.AnnotatedCmd as the main touch point for users, I've included a big docstring there including this information which should make it easy to discover for users of bk.AnnotatedCmd

Closes https://github.com/sourcegraph/sourcegraph/issues/26071

How to use it

In your script, leave a file in ./annotations:

if [ $EXIT_CODE -ne 0 ]; then
  echo -e "$OUT" >./annotations/docsite
fi

In your pipeline operation, replace bk.Cmd with bk.AnnotatedCmd:

	pipeline.AddStep(":memo: Check and build docsite",
		bk.AnnotatedCmd("./dev/check/docsite.sh", bk.AnnotatedCmdOpts{
			Type: bk.AnnotationTypeError,
		}))

That's it!

See https://buildkite.com/sourcegraph/sourcegraph/builds/130786:

Big screenshot

Rules of the game:

• You either create a MultiJobContext, or all annotations from a job go into a single annotation block.
• For both MultiJobContext and default per-job annotations, annotations of different levels go into separate annotation blocks
• Limited formatting options: content is either directly appended into annotation, or we include the name of the annotation file and include that as a heading.
  • .md files in annotations are treated as markdown content, otherwise we format them as terminal content

How it works

1. annotated-command.sh, aliased as ./an, turns pipeline options into arguments and runs the given command
2. Once the command has run, we then scan the ./annotations directory for files
3. Pass each file to annotate.sh with the generated arguments and some other parameters based on the name of the file, e.g. if it has a .md extension or if we should include the file name.
4. Done!

Benefits:

• no more enterprise scripts being called in your check scripts
• no more checking of $BUILDKITE or buildkite-agent
• view annotations locally
• easily see which jobs generate annotations

Test plan

This PR is on a main-dry-run that runs everything. https://github.com/sourcegraph/sourcegraph/commit/7dafb63f601e39e025b181a829969a2abaa95af1 triggers intentional linter errors that will generate annotations. Security scanning annotations are always around, so we verify if that works.