ranking: flush result queue after X ms

Created by: stefanhengl

We flush the result queue after maxReorderDuration ms. The default is 0 which maintains the current behavior.

For now this is just an experiment to better understand the dynamic of the result queue.

Why?

So far we flush the queue if

(1) all pending results have lower priority, or (2) the queue is full

For searches with very common terms, the result queue fills up quickly, sometimes giving us not enough time to search some of the most popular, larger repositories.

Example: The shard size of golang/go is around 200mb. A search with common terms, such as "http" and "get", yields results from a lot of repositories, most of which are tiny. The result queue fills up before we finished searching golang/go. Therefore results from golang/go frequently don't show up if searches are too broad.

I confirmed this scenario by splitting golang/go in many smaller shards. Once the shards were smaller, their results showed up in Sourcegraph. However, increasing the number of shards comes at a cost of MEM and DISK for Zoekt.

Note: We could simply increase the queue length until results from golang/go show up. However, setting a duration is a much more direct measure that translates to a budget per Zoekt instance instead of to an overall budget shared by all instances.

Experiments on .com indicate that a reasonable value for maxReorderDuration could be around 50ms.

Test plan

• existing tests
• the default setting is a NOP
• local testing, setting various values of maxReorderDuration