Skip to content
Snippets Groups Projects
Closed ssbc: Implement basic fairness for dequeueing workspaces
  • View options
  • ssbc: Implement basic fairness for dequeueing workspaces

  • View options
  • Closed Issue created by Warren Gifford

    Created by: eseliger

    We currently see a problem with users clogging the executors queue when they start a batch change execution with a lot of workspaces. Once they add 1000 jobs, they all need to be worked off before another user can get a single workspace executed.

    To work around this, we want to implement fairness in the executor worker. To achieve this, we can use a SQL view for the dbworker backing the executor that does some custom ordering. We will also need to make sure that the position in queue is returned correctly from the API. For this field, we want to return the place in the user-specific queue, instead of the global queue. Users should not be concerned about the global rank, since a healthy, autoscaling executors setup should make sure that no user has to wait. This is not implemented yet, but a worry for later.

    This implements a part of https://github.com/sourcegraph/sourcegraph/issues/26895, which is the subset we need for SSBC beta. The other considerations from that ticket don't need to be considered for now. They are mostly relevant on a shared multi-tenant environment.

    I did some experiment previously that sort of worked but isn't perfect: It utilizes a window function to intertwine the results of multiple user queues.

    ORDER BY
    	-- Round-robin let tenants dequeue jobs
    	ROW_NUMBER() OVER (
    		PARTITION BY q.user_id
    		ORDER BY
    			-- Still dequeue in order of creation (todo: readd order by crated_at, process_after)
    			j.id,
    			-- TODO: Doesn't work. This should enforce that user 2 can dequeue a job first.
    			q.latest_dequeue ASC NULLS FIRST
    	);

    Tasks:

    2 of 4 checklist items completed

    Activity

    • All activity
    • Comments only
    • History only
    • Newest first
    • Oldest first