Skip to content

Compute: process in parallel

Warren Gifford requested to merge cc/pargroup into main

Created by: camdencheek

This runs compute post-processing in parallel.

This PR has two commits:

  1. Create the new group.OrderedParallel type. This allows us to run tasks in parallel but still process the results in the same order they were submitted.
  2. Update compute code to use group.OrderedParallel. This would conflict with in-flight work, so this PR is stacked on #34230 (closed)

Also, I noticed that for the content:output(...) compute filter, we are fetching the result content for every file match even if we don't use it (e.g content:output(test -> $lang)). For queries like that, it would be way more efficient to skip fetching content. Though, solving the multiline matches also fixes this, so I'm probably not going to focus on that immediately unless it means we can get rid of the stats endpoint sooner.

I won't merge until the PR this is stacked on is merged.

Related to https://github.com/sourcegraph/sourcegraph/issues/35500

Test plan

Added tests for group.ParallelOrdered. Also tested perf improvements manually. See video

Merge request reports