Skip to content

codeintel: Add retries to larger bodies in bundle manager client

Administrator requested to merge bundle-manager-retries into master

Created by: efritz

The bundle manager layer already performs retries for downloading upload payloads from the bundle manager. There is no current retry mechanism for uploading an upload or database part, which means if one part in a multipart upload fails, the entire thing fails (which can still happen due to k8s-proxy issues and due to services restarting on deployments).

This PR adds a retry mechanism in order to re-attempt upload requests in the bundle manager layer. The technical outline is:

  • client accepts reader to post to bundle manager
  • client tees the reader into a local file and also an http request
  • on failure, subsequent requests are made by sending (file content = remaining reader) as the next http requset

I'm not sure what good values for max retry delay/attempts are here, and I think it likely changes based on deployment environments.

Most of the transient errors in https://github.com/sourcegraph/sourcegraph/issues/10913 were taken care of by https://github.com/sourcegraph/sourcegraph/pull/11004, but this will increase the robustness of the paths that are mostly fixed with a few occasional errors slipping through.

Merge request reports