Skip to content

LSIF: Unify dump and upload models.

Administrator requested to merge lsif-unify-uploads-and-dumps into master

Created by: efritz

This is part one of a change to unify LSIF dumps and uploads. The current plan:

  1. Unify the tables in the database into one table, ensuring that there is no regression in the query hot-path when adding additional records into the same tablespace (this PR).
  2. Collapse LSIF dump and upload endpoints in the lsif-server and in the GraphQL API. LSIF endpoints are still experimental so we have a lot of flexibility here.
  3. Remove the site-admin LSIF uploads page and add additional context to each repo's code intelligence page. All other data that a site-admin would care about can be captured in metrics.

This PR should be reviewed by commit, which:

  1. Create a migration to add additional fields of lsif_uploads into the lsif_dumps table (keeping all of the original lsif_dump rows). This migration also moves all upload data into the lsif_dumps table, then drop lsif_uploads and renames lsif_dumps to lsif_uploads. This adds a few new non-nullable fields to the existing table.
  2. Create a view lsif_dumps which is a query of lsif_uploads with the state "completed". This is basically all that differentiated lsif_dumps and lsif_uploads previously. This makes the following changes to the code minimal.
  3. Update the worker process, which used to dequeue an upload, create a dump, and mark the dequeued upload as finished. Now, it must dequeue an upload and update the same record with additional job and dump metadata.
  4. Ensure that the last step is done transactionally. It was easier to get away with less strict transactions when operating over retryable jobs. This requires threading entity managers through some methods that didn't require a transaction previously.

Merge request reports

Loading