ci: Improve performance of go build for server
Created by: keegancsmith
Demo video: https://www.loom.com/share/db21ed115d3e4e78ba8eddab7c6b52a0
Older versions of go may have been more unreliable. But in this case we should be able to rely on the build cache. Additionally this was the only image we set -a. An example CI job had the building of go images take 3m15s. After this change a go build took 28s.