Bare-metal Buildkite agents capable of running Docker and VMs

Created by: slimsag

Quoting the Distribution roadmap:

Today we cannot release of Sourcegraph, run e2e tests, or perform Docker image tests in an automated fashion because our CI infrastructure does not support running Docker containers (or VMs/Vagrant) in a reliable way. Today we have a side-car DIND container in our CI pipeline but it is flaky, unreliable, and a regular source of issues which has led to us removing automated testing (see Automatic e2e testing).

I would like to see us have some sort of CI infrastructure that is capable of running:

• Docker containers
• Docker Compose
• Vagrant

I propose we create bare-metal Buildkite agents to solve this issue, provisioned through Terraform. This would then allow us to do a number of things:

• Have confidence that our e2e test failures are NOT due to the sidecar dind node, and therefor run e2e tests in CI once again.
• Perform automatic Docker image testing
• Automate the Docker Compose release process (which requires e.g. running docker-compose up -d and doing basic health checks on specifically a Linux host OS)

TODO

• Build some tooling so make the process of building the image repeatable with something simple like make build that can be run locally.