ci: Docker In Docker sidecar dying will led to a failed build and not a canceled one.
Created by: jhchabran
This failed because of docker daemon going down:
- I redeployed agents
- at the same time was ran https://buildkite.com/sourcegraph/sourcegraph/builds/111980
- failed because the Docker Daemon was not responding
- I suspect that the DinD sidecar shut down before the agent.
- edit: Reading about pod terminationmakes me think that's probably because the DinD shutdowns faster than the agent.
The big problem here is that the build is a failure, when in reality it is being canceled. This will mess with observability if unaddressed.
Big thanks to @mrnugget for taking the time to jump and pair on his failed build to have a closer look.