Skip to content

codeintel: Add an upload store API layer

Warren Gifford requested to merge ef/block-storage-api into main

Created by: efritz

This closes https://github.com/sourcegraph/sourcegraph/issues/14823.

This creates a persistence layer that will eventually replace the bundle manager's final remaining task: uploading and downloading byte ranges of an LSIF upload. This persistence layer works with

  • GCS, given credentials in the Google Way ™️
  • S3, given credentials in the Amazon Way ™️
  • MinIO, given credentials similar to S3 (requires additional vars AWS_ENDPOINT and S3_FORCE_PATH_STYLE)

For each of these layers, I've hand-verified that:

  • the configured bucket is created on client creation if it doesn't exist
  • the library does not freak out if the bucket already exists (the APIs for GCS and S3 are asymmetric here)
  • each bucket is created with an expiration lifecycle so that objects older than the configured age are automatically deleted
  • the expiration lifecycle is recreated if it's deleted or changed (this allows us to change the expiry configuration at runtime without operator intervention)
  • buckets are created with sane default (non-public) ACLs
  • file content can be uploaded and read back (in full and in chunks)

Unfortunately I found these hard to unit-test, as we're really just writing a compatibility layer on top of a pair of libraries. We could mock them out but I don't know what we get out of that. All of the configuration code was tested (and did find a reference/value-type bug!).

Tips for this are welcome, but I'm confident of the current code.

Merge request reports