httpcli: add retries to NewExternalHTTPClientFactory
Created by: tsenart
This commit adds an error resilience transport middleware to the
NewExternalHTTPClientFactory
. This means that in every place we
talk to the outside world via HTTP, we'll now automatically retry
requests that meet the criteria defined in DefaultRetryPolicy
.
To support retries, we increase the default timeout to 5 minutes.
This change is driven by intermittent failures in long external service syncing jobs that cause us to entirely fail the whole thing, when we can instead retry sporadic errors and deploys of github-proxy that happen frequently.