Skip to content

uploadstore: Manually expire objects when using MinIO

Warren Gifford requested to merge ef/19729 into main

Created by: efritz

Fixes https://github.com/sourcegraph/sourcegraph/issues/19729.

Changes in behavior:

  • No changes to (externally configured) S3 or GCS buckets. We can continue to use those lifecycle configuration endpoints and they will work just fine.
  • If MinIO is configured, then we don't send a lifecycle configuration update request (it ignores it). Instead, we'll periodically scan the store for expired objects and delete them.

Changes in code:

  • If MinIO is configured, we wrap the existing S3 API in a decorator that switches behavior of PutBucketLifecycleConfiguration
  • The default (externally configured S3) behavior that configure the bucket's lifecycle configuration has moved from the "client" layer (that has application behavior) to the "api" layer (that has backend S3/GCS-specific behavior)
  • The decorated lifecycle configuration method (new) starts two goroutines to periodically read and delete expired objects concurrently. This will periodically re-scan the object store every 30 minutes (+/-5 minutes so concurrent scans are less likely). I don't see a need to configure this.

Merge request reports