Skip to content

codeintel: Group code intel data for serialization on-demand

Administrator requested to merge ef/pci-group-channels into main

Created by: efritz

This changes the grouping layer that converts correlated raw LSIF data into the format that is written to disk from being pre-computed to being computed when needed by the batch writer.

This saves us from grouping everything in memory all at once (which requires that we have both the correlated data in memory, AND the grouped data before beginning to write it to disk). We now correlate everything in memory, and return only the batches that are going to be immediately serialized, keeping our resident memory down.

This will help us with concurrent serialization, as more processors can run at once with a lower memory requirement.

Merge request reports

Loading