Search backend: convert and/or evaluation to jobs
Created by: camdencheek
This rewrites our and/or pattern evaluation code to take full advantage of streaming and to generate jobs for them. This code is all behind the feature flag cc-and-or-jobs
.
A couple of highlights:
- Both
And
andOr
queries now stream whenever possible. "Whenever possible" in this case means "whenever a match has been streamed by every child job" When that happens, we know it's safe to forward on a result, otherwise we may miss merging some matches. - Everything up to the
r.evaluate()
call is "statically" described by jobs. This is huge progress, and leaves only one more layer to convert to jobs (r.resultsRecursive()
) - Massive simplification
- I think this opens up the opportunity to optimize our job tree to send zoekt-native
and
queries, which would be an orders-of-magnitude improvement over the current situation
I expect there to be some "feeling out" how well this works at the scale of sourcegraph.com
, including but not limited to:
- Imposed limits
- Tradeoffs between correctness/performance
- When to alert
Comments inline.