Skip to content

Expiration of failed upload and index records

Created by: efritz

We currently expire completed upload records to remove large associated data from the codeintel-db. We have no data retention on failed upload records, or data retention on index metadata records (at all). The lsif_upload tables can become quite big, which is bad for some classes of SQL queries that currently use table scans.

We should add a background job that removes lsif_upload records that:

  • have failed
  • are older that some configurable threshold (default, say, ~1 month?)
  • are not the last record of its type (for a repo/root)

We should also add a background job that removes lsif_index records in a similar fashion.