Created by: tbliu98
This PR builds on the changes in https://github.com/sourcegraph/sourcegraph/pull/37083 to implement streaming into and out of comby
for indexed structural search, which noticeably improves performance. These changes depend on this comby commit which should be included in the next release (current version as of writing is 1.7.1).
Key changes:
TarInput
as an implementor of comby.Input
-tar
flag (commit linked above). Introduce the type comby.FileMatchWithChunks
as the unmarshalled format of comby's output when returning chunk matches is active.zoektSearch()
now calls StreamSearch()
with a callback function that converts each candidate match to tar format and sends it to a channel. structuralSearch()
receives on that channel and blocks until it's finished receiving.protocol.FileMatch
es as output is read from comby's stdout.FYI The original plan was to also include streaming results from comby for unindexed structural search, but I haven't gotten that working yet. It might make sense to treat unindexed search as its own issue and tackle it in a separate PR stacked on this one, especially since merging this PR is blocked until the next comby release anyways.
Added test for streaming tar input to comby via stdin. Manually tested indexed structural search. Ran backend integration tests https://buildkite.com/sourcegraph/sourcegraph/builds/156098