Skip to content

Packages: reduce number of GET requests to sync packages

Administrator requested to merge olafurpg/faster-package-sync into main

Created by: olafurpg

Previously, we sent a large number of requests to sync packages. My rough estimate is that we sent 3 GET requests for every tracked package+version combination. After this PR, we send only one GET request for every tracked package+version, and for npm we additionally send one GET request per package (regardless of number of versions) in order to fetch the package description.

This PR also includes a few more renames that should have been part of #37999

  • rename "go modules" into "go packages" (for consistency with all other package implementations, including "jvm packages" even if JVM package is similar to Go package)
  • rename PackageVersion into VersionedPackage, fixes https://github.com/sourcegraph/sourcegraph/issues/38009

Fixes https://github.com/sourcegraph/sourcegraph/issues/37990

Test plan

See the CI go green. I manually tested the changes locally by syncing 120k crate versions in less than one hour. For comparison, it took us ~30hr on sourcegraph.com to sync the same number of crate versions.

Merge request reports