Tracking: release rough edges

Created by: bobheadxi

Mini-tracking issue for some rough edges encountered by @pecigonzalo , most notably during the recent 3.22 release (#15023 (closed)). Adapted from a Google doc he wrote up. We should split some of these into separate issues if they amount to a lot of work.

Check list indicates if an issue was created to track work, or if the problem itself was resolved directly

Release tool

• If you dont enter a version on upcoming version (prompt) it crashes, and you cant recover, you have to git clean -fxd => bug in confirmation prompt that ensures captains are aware of the configured version => fixed in https://github.com/sourcegraph/sourcegraph/pull/16689
• Secrets configuration could be from something more universal (.env? .envrc?) => opened issue https://github.com/sourcegraph/sourcegraph/issues/16645
• Would be nice to have a more "CLI-like" experience => fixed in https://github.com/sourcegraph/sourcegraph/pull/16519
• Terrible things happen if release tool is interrupted (e.g. error on a repo) => opened issue https://github.com/sourcegraph/sourcegraph/issues/16549
• Errors leave release in bad state => https://github.com/sourcegraph/sourcegraph/issues/16549

Continuous integration

• env vars in CI env are not good because they are displayed directly in the console and stored at Buildkite. => opened issue https://github.com/sourcegraph/sourcegraph/issues/16647

Incorrect documentation

• git branch 3.22 incorrect => fixed in https://github.com/sourcegraph/about/pull/2128
• "Upon branch cut, create and test the first release candidate:" should indicate to checkout 3.22 this step is not required, candidates are cut from branches by the release tool
• yarn run release release-candidate:create final should be yarn run release release:create-candidate final => fixed by https://github.com/sourcegraph/about/pull/2128
• Many steps on https://github.com/sourcegraph/deploy-sourcegraph-docker/blob/master/RELEASING.md are already done by buildkite, should note and/or remove => fixed in https://github.com/sourcegraph/deploy-sourcegraph-docker/pull/192
• Testing instructions => covered by #15750 (closed) , instead of fixing this I believe we should just remove these steps
  • Regression tests docs point to wrong directory, should be client/web, tools missing
• Releasing pure-docker are incorrect => opened issue https://github.com/sourcegraph/sourcegraph/issues/16646

Tracked issues

@bobheadxi: 0.50d

• release: continue on errors (#16549) 0.50d

Completed

• (🏁 1 day ago) release: remove manual regression testing from release process (#15750)