Introduced a "Failed" state to dbworker and reconciler
Created by: mrnugget
This ticket is the result of a discussion I had with @eseliger about this snippet of code here:
It was introduced in https://github.com/sourcegraph/sourcegraph/pull/15528 and we both said that we should make the behaviour of "terminal errors" and "failed state" more explicit so that ideally this snippet of code disappears.
Plan:
- Introduce another dbworker/reconciler state that is "failed", which is a job that is errored (and has a failure message) but shouldn't be retried
- Change the dbworker logic so that when a worker handler returns a "terminal error" (one which can't be retried) the transaction is not rolled back, but instead the record is marked as failed and the failure message is set
- Remove the
if err.IsTerminal(err)fromexecutorand simply bubble up the error.