Skip to content

ssbc: Implement basic fairness for dequeueing workspaces

Created by: eseliger

We currently see a problem with users clogging the executors queue when they start a batch change execution with a lot of workspaces. Once they add 1000 jobs, they all need to be worked off before another user can get a single workspace executed.

To work around this, we want to implement fairness in the executor worker. To achieve this, we can use a SQL view for the dbworker backing the executor that does some custom ordering. We will also need to make sure that the position in queue is returned correctly from the API. For this field, we want to return the place in the user-specific queue, instead of the global queue. Users should not be concerned about the global rank, since a healthy, autoscaling executors setup should make sure that no user has to wait. This is not implemented yet, but a worry for later.

This implements a part of https://github.com/sourcegraph/sourcegraph/issues/26895, which is the subset we need for SSBC beta. The other considerations from that ticket don't need to be considered for now. They are mostly relevant on a shared multi-tenant environment.

I did some experiment previously that sort of worked but isn't perfect: It utilizes a window function to intertwine the results of multiple user queues.

ORDER BY
	-- Round-robin let tenants dequeue jobs
	ROW_NUMBER() OVER (
		PARTITION BY q.user_id
		ORDER BY
			-- Still dequeue in order of creation (todo: readd order by crated_at, process_after)
			j.id,
			-- TODO: Doesn't work. This should enforce that user 2 can dequeue a job first.
			q.latest_dequeue ASC NULLS FIRST
	);

Tasks: