Skip to content

LSIF: Prune old dumps

Administrator requested to merge lsif-retention into master

Created by: efritz

This implements a superset of functionality as discussed in RFC 55.

Relevant changes:

  • Moved databases into a subdir dbs in the storage root. This is so we can iterate the entire directory without worrying about walking/omitting the temp or upload files. There is an automatic migration that occurs on the startup of the server.
  • Create a new scheduled job type, clean-failed-jobs, that will remove all files in the uploads and temp directories if they are older than a configurable maximum age. This will delete files that stuck around from a failed conversion. This also assumes that the maximum age is greater than the time it takes to process an LSIF dump upload (from the time the upload is accepted until the time the job has completed).
  • After each successful upload, delete the oldest non-visible dumps (as discussed in the RFC) while the space occupied by dumps is above a configurable maximum. To reduce issues with two workers trying to clean the same directory concurrently, this process happens while holding a postgres advisory lock.

This closes #6148.

Merge request reports

Loading