ci: caching mechanism for stateless agents

Created by: jhchabran

Follow-up of https://github.com/sourcegraph/sourcegraph/pull/30123

Fixes: #21130 #30234 (closed) and #30235 (closed)

This adds support for caching to improve build times on stateless agents. Without it, the build times are so high that it challenges the whole idea of going stateless.

It caches the biggest drags of moving to stateless agents (Docker excluded)

• Node modules

  • about 200s saved on every build involving node
  • that can be taken further by working with yarn specifics, but it requires investing more time.

• asdf

  • about 20s saved on every builds
  • less external HTTP calls

• Go modules Cache will fix

  • Need to enable the cache plugins to deal with absolute paths, postponed for the moment, should just be a matter of patching the plugin for those specific cases.

Other changes:

• If WANT_STATELESS is set to true, the pipeline generator will force a build of the MainDryRun type, targeting the job queue.
• Fixed the Plugin function that allowed to use of only a single plugin
• Added docs enabling everyone to make use of the cache.
• Updated the pre-command hook to use asdf caching if and only if we're running on a stateless agent

➡️ Once this is merged, we can set up a scheduled build that takes the latest main and builds it on the stateless agents, let's say every hour. From there, we can observe its behavior until we are confident enough to roll it out.