Skip to content

Stream based search API

Created by: keegancsmith

This issue is to track our work towards streaming search. Our aim is two-fold:

  1. Return results as soon as available.
  2. Exhaustive large result sets don't OOM any services or timeout.

These aims imply that we have better handling around searching unindexed repos. IE if a repo is not yet available for searching, this is communicated in a much smoother way and does not block the return of indexed search.

The main goal of the 3.20 iteration is streaming results into the webapp. We may want to do further changes where the RPCs between sourcegraph-frontend and zoekt/searcher are stream based (but are not tied to this goal). This would further improve the experience.

We currently have a PoC working just in Zoekt using Server Sent Events. This validated the approach of how to structure parts of our Go code as well as exposing the API via server sent events. Our current goal is to add a similiar API to sourcegraph-frontend for use by the webapp.

We considered graphql subscription and could still use them. However, they require websockets and this use case doesn't exactly match the need. We would also need to implement it for our graphql-go, but the implementation looks relatively straight forward.