Improve performance of LSIF upload processing
Created by: creachadair
Converting LSIF uploads into SQLite databases can be a bottleneck for indexing large repositories. Processing a large upload can cause worker processes to exceed their RAM budget and terminate, and throttle or starve the processing queue.
Upload conversion currently runs on Node, mainly for historical reasons. As a step toward understanding and improving the performance of the conversion, we are going to start by re-implementing the convertLsif function that handles the core conversion logic in Go. This will make it easier for us to hook up CPU and memory profiling, without disrupting the existing worker logic too much.
As a temporary step, the worker will communicate with the converter via a pipe. Later, if this is fruitful, we'll investigate replacing more of the worker and remove that dependency.