codeintel: Add a queue for dependency indexing
Created by: efritz
Replace the QueueIndexesForPackage
call from the worker. Instead, the worker will create a new queued job record in a new queue processed by a worker in the frontend (we may move this in the near future). These new records are attached directly to an lsif_upload record so do not need to be cleaned periodically like other records may (deletion of one is followed by the other).
Fixes https://github.com/sourcegraph/sourcegraph/issues/20838.
Future tasks:
-
Add dashboards (some metrics already being emitted, should add additional) https://github.com/sourcegraph/sourcegraph/issues/21606 -
Add a way to de-hard-code the list of accepted repository IDs (a simple db table is a good start) https://github.com/sourcegraph/sourcegraph/issues/21607 -
Add a way to re-trigger dependency scanning for a particular index https://github.com/sourcegraph/sourcegraph/issues/21608 -
Add way to show job progress in the UI (with links to spawned index jobs) https://github.com/sourcegraph/sourcegraph/issues/21609 -
Perform ReferencesForUpload in batches and log progress https://github.com/sourcegraph/sourcegraph/issues/21610 -
Perform QueueIndexesForPackage transactionally https://github.com/sourcegraph/sourcegraph/issues/21611