LSIF: Prune old dumps
Created by: efritz
This implements a superset of functionality as discussed in RFC 55.
Relevant changes:
- Moved databases into a subdir
dbs
in the storage root. This is so we can iterate the entire directory without worrying about walking/omitting the temp or upload files. There is an automatic migration that occurs on the startup of the server. - Create a new scheduled job type, clean-failed-jobs, that will remove all files in the uploads and temp directories if they are older than a configurable maximum age. This will delete files that stuck around from a failed conversion. This also assumes that the maximum age is greater than the time it takes to process an LSIF dump upload (from the time the upload is accepted until the time the job has completed).
- After each successful upload, delete the oldest non-visible dumps (as discussed in the RFC) while the space occupied by dumps is above a configurable maximum. To reduce issues with two workers trying to clean the same directory concurrently, this process happens while holding a postgres advisory lock.
This closes #6148.