Skip to content

codeintel: Run workers concurrently

Administrator requested to merge concurrent-workers into master

Created by: efritz

Support the worker processing multiple uploads concurrently.

Bug fixes:

  • The dequeueRecord function in the code intel store was subtly broken: we tried to SELECT FOR UPDATE a view, which restrained possible concurrency by locking the entire underlying table. This has been fixed by ensuring we only run this query on the actual table and not the view. This will be cleaned up additionally by https://github.com/sourcegraph/sourcegraph/issues/11786.

New functionality:

  • Added (nullable) upload_size column to lsif_uploads table
  • The bundle manager upload endpoints now return the size of the file on disk as a response payload
  • The bundle manager client now reads this for the terminal requests (send upload and stitch uploads)
  • The HTTP LSIF upload proxy writes the upload size to the upload record when setting the state to queued
  • The codeintel store now accepts a maxSize for the dequeue method
  • The worker processes upload records in goroutines. The number of goroutines are limited by a counting semaphore implemented as a channel. The work accepted by goroutines are limited by selecting only upload sizes less than the current worker budget.

This closes https://github.com/sourcegraph/sourcegraph/issues/11643.

Merge request reports

Loading