Skip to content

Search backend: convert and/or evaluation to jobs

Warren Gifford requested to merge main-dry-run/cc/and-or-jobs into main

Created by: camdencheek

This rewrites our and/or pattern evaluation code to take full advantage of streaming and to generate jobs for them. This code is all behind the feature flag cc-and-or-jobs.

A couple of highlights:

  • Both And and Or queries now stream whenever possible. "Whenever possible" in this case means "whenever a match has been streamed by every child job" When that happens, we know it's safe to forward on a result, otherwise we may miss merging some matches.
  • Everything up to the r.evaluate() call is "statically" described by jobs. This is huge progress, and leaves only one more layer to convert to jobs (r.resultsRecursive())
  • Massive simplification
  • I think this opens up the opportunity to optimize our job tree to send zoekt-native and queries, which would be an orders-of-magnitude improvement over the current situation

I expect there to be some "feeling out" how well this works at the scale of sourcegraph.com, including but not limited to:

  • Imposed limits
  • Tradeoffs between correctness/performance
  • When to alert

Comments inline.

Merge request reports

Loading