RFC ci: Only E2E run for releases
Created by: keegancsmith
I expect this to be a bit controversial. This is a request for comment in the form of a PR. Please state your opinions as comments. cc @sourcegraph/code-intel @sourcegraph/core-services @sourcegraph/distribution
E2E test reliability only seems to be getting worse. Just today I have had to hit retry 3 times on builds I landed in master. This has been an ongoing issue, and I believe we have reached the point where running E2E on PRs/master provides more bad than good. This is one possible short-term solution.
PRs and master will no longer run E2E tests. This should improve reliability and speed of CI for the general case, while still ensuring e2e tests pass on tagged releases.
The downside is we won't notice severe regressions to e2e, so will have to deal with the issues once we do a release. But as it stands the e2e tests are in a bad shape and no one is fixing them. We are building a pavlovian response to e2e failing and the retry button.