codeintel: LSIF line reader
Created by: efritz
Refactor the function that reads and unmarshalls lines of LSIF data from a source. This is done in a generic (focus on reading, not unmarshalling) way so we can replace/supplment JSON as an encoding in the future.
There are three significant changes:
- Add a buffered channel between the bufio Scanner and the consumer so that IO can be done ahead of the consumer
- Introduce a pool of unmarshal workers that can work on a batch of lines concurrently
- Buffer the output channel so that unmarshalling can be done ahead of the correlator process
Running the precise-code-intel integration suite's upload phase on master:
3m37.597s
3m35.858s
3m42.591s
And after this change:
2m29.395s
2m32.812s
2m30.429s