insights: reduce overhead of queue operations on query throughput

Created by: coury-clark

Code Insights uses a queue to asynchronously process search queries required to backfill or record insights values. Each record is dequeued individually, then the search query is executed and processed. Execution time of the search query scales with the size of the response, but the overwhelming number of queries executed will be repo-specific queries dedicated to backfilling a series. These queries typically execute very fast, with a p99 that hovers around 100ms on dogfood.

When we look at the queue operations in isolation, we can see a substantial amount of time is being spent on queue operations.

Ultimately the result is that insights is throughput limited by the queue operations (currently hovering around 10 qps). Ideally we strive to be throughput limited by search operations.

A few approaches:

1. Support batch dequeue operations (see https://github.com/sourcegraph/sourcegraph/pull/24297 for some thoughts)
2. Support concurrency in the queue orchestration

The scope of this issue is broad, so we will loosely define some target metrics that would be considered acceptable for the time being:

1. Reach the currently configured rate limit of 40 qps
2. Ideally reduce the amortized queue operation cost below the p99 execution time (~100ms)