Skip to content

Structural search: Stream to and from comby for indexed structural search

Administrator requested to merge tl/stream-structural into main

Created by: tbliu98

This PR builds on the changes in https://github.com/sourcegraph/sourcegraph/pull/37083 to implement streaming into and out of comby for indexed structural search, which noticeably improves performance. These changes depend on this comby commit which should be included in the next release (current version as of writing is 1.7.1).

Key changes:

  • Introduce TarInput as an implementor of comby.Input
  • comby now supports returning chunk matches for invocations with the -tar flag (commit linked above). Introduce the type comby.FileMatchWithChunks as the unmarshalled format of comby's output when returning chunk matches is active.
  • zoektSearch() now calls StreamSearch() with a callback function that converts each candidate match to tar format and sends it to a channel. structuralSearch() receives on that channel and blocks until it's finished receiving.
  • On the indexed search path, write candidate matches to comby's stdin as they are received from the channel, and stream out protocol.FileMatches as output is read from comby's stdout.

FYI The original plan was to also include streaming results from comby for unindexed structural search, but I haven't gotten that working yet. It might make sense to treat unindexed search as its own issue and tackle it in a separate PR stacked on this one, especially since merging this PR is blocked until the next comby release anyways.

Test plan

Added test for streaming tar input to comby via stdin. Manually tested indexed structural search. Ran backend integration tests https://buildkite.com/sourcegraph/sourcegraph/builds/156098

Merge request reports