Expiration of failed upload and index records
Created by: efritz
We currently expire completed upload records to remove large associated data from the codeintel-db. We have no data retention on failed upload records, or data retention on index metadata records (at all). The lsif_upload tables can become quite big, which is bad for some classes of SQL queries that currently use table scans.
We should add a background job that removes lsif_upload records that:
- have failed
- are older that some configurable threshold (default, say, ~1 month?)
- are not the last record of its type (for a repo/root)
We should also add a background job that removes lsif_index records in a similar fashion.