Skip to content

codeintel: LSIF line reader

Administrator requested to merge lsif-line-reader into master

Created by: efritz

Refactor the function that reads and unmarshalls lines of LSIF data from a source. This is done in a generic (focus on reading, not unmarshalling) way so we can replace/supplment JSON as an encoding in the future.

There are three significant changes:

  • Add a buffered channel between the bufio Scanner and the consumer so that IO can be done ahead of the consumer
  • Introduce a pool of unmarshal workers that can work on a batch of lines concurrently
  • Buffer the output channel so that unmarshalling can be done ahead of the correlator process

Running the precise-code-intel integration suite's upload phase on master:

3m37.597s
3m35.858s
3m42.591s

And after this change:

2m29.395s
2m32.812s
2m30.429s

Merge request reports

Loading