codeintel: Improve stitching
Created by: efritz
This PR updates the codeintelutils to provide a parameter to control file stitching compression of the resulting file / decompression of each parts. The upload stitch endpoint is a bit slow as it needs to read all part files, decompress them, concatenate them, then recompress them into the target file.
This is unnecessary as the concatenation of a gzip stream is a valid gzip stream. We don't need to de/re-compress the stream and can simply concatenate the files. (This is not true of db files, which are uncompressed on disk so that sqlite can read them).
This PR additionally cuts out the tar archive/extraction for database files, as we're only ever sending one file and wrapping it in a tar archive is unnecessary. This change was made when we were exploring alternate backends that are not restricted to a single file.
This covers the second task in https://github.com/sourcegraph/sourcegraph/issues/11876.
Merge request reports
Activity
Created by: codecov[bot]
Codecov Report
Merging #11958 into master will increase coverage by
11.11%
. The diff coverage is50.00%
.@@ Coverage Diff @@ ## master #11958 +/- ## =========================================== + Coverage 36.60% 47.71% +11.11% =========================================== Files 720 1412 +692 Lines 21930 80158 +58228 Branches 6811 6757 -54 =========================================== + Hits 8027 38246 +30219 - Misses 13844 38334 +24490 - Partials 59 3578 +3519
Flag Coverage Δ #go 51.89% <50.00%> (?)
#storybook 10.75% <ø> (ø)
#typescript 36.60% <ø> (ø)
#unit 47.29% <50.00%> (+12.23%)
Impacted Files Coverage Δ ...e-code-intel-indexer/internal/indexer/processor.go 0.00% <0.00%> (ø)
...ise-code-intel-worker/internal/worker/processor.go 45.79% <0.00%> (ø)
.../codeintel/bundles/client/bundle_manager_client.go 70.14% <100.00%> (ø)
internal/httputil/cache_control_transport.go 90.90% <0.00%> (ø)
enterprise/internal/campaigns/changeset_events.go 95.45% <0.00%> (ø)
enterprise/internal/codeintel/api/definitions.go 59.57% <0.00%> (ø)
internal/conf/validate_custom.go 28.57% <0.00%> (ø)
internal/extsvc/bitbucketserver/client.go 50.29% <0.00%> (ø)
cmd/frontend/graphqlbackend/search_repositories.go 80.68% <0.00%> (ø)
cmd/frontend/internal/auth/forbid_all.go 100.00% <0.00%> (ø)
... and 685 more