Skip to content

codeintel: Improve stitching

Warren Gifford requested to merge pci-detar into master

Created by: efritz

This PR updates the codeintelutils to provide a parameter to control file stitching compression of the resulting file / decompression of each parts. The upload stitch endpoint is a bit slow as it needs to read all part files, decompress them, concatenate them, then recompress them into the target file.

This is unnecessary as the concatenation of a gzip stream is a valid gzip stream. We don't need to de/re-compress the stream and can simply concatenate the files. (This is not true of db files, which are uncompressed on disk so that sqlite can read them).

This PR additionally cuts out the tar archive/extraction for database files, as we're only ever sending one file and wrapping it in a tar archive is unnecessary. This change was made when we were exploring alternate backends that are not restricted to a single file.

This covers the second task in https://github.com/sourcegraph/sourcegraph/issues/11876.

Merge request reports

Loading