Skip to content

executor: Extract executor-queue from precise-code-intel-indexer

Warren Gifford requested to merge ef/extract-executor-queue into main

Created by: efritz

Note: There are actually only 886 SLOC in this PR. The rest are test, mocks, and dev plumbing scripts.

This PR extracts the relevant behaviors from the precise-code-intel-indexer service which is not specific to the scheduling of code intelligence jobs.

The general idea is to have a generic queue API with the following endpoints:

  • /{queueName}/dequeue attempts to select and lock a row in the database within a transaction which is held for a long time. This job is returned and all subsequent interaction with the queue for this job will specify its ID in the request body.
  • /{queueName}/setLogContents sets the log contents for a job record being held by the client.
  • /{queueName}/markCompleted marks the job record being held by the client as complete and commits the txn.
  • /{queueName}/markErrored marks the job record being held by the client as errored and commits the txn.
  • /{queueName}/heartbeat provides a list of jobs being processed by the executor so that we can reconcile orphaned jobs on the executor side and prevents us from holding transactions for job dequeued by executors which have become unresponsive.

In order for this to be fully generic, we define a Job type, and install a transformer on the response side of the API specific to a queue. This will allow codeintel, campaigns, and other future users to transform some type of rich database object into a generic job which can be understood by the downstream completely-generic executor process.

This closes https://github.com/sourcegraph/sourcegraph/issues/14834.

Merge request reports