Skip to content

Download crates from CDN instead of crates.io API

Warren Gifford requested to merge olafurpg/crates-cdn into main

Created by: olafurpg

Fixes https://github.com/sourcegraph/sourcegraph/issues/38559

Previously, we downloaded crate contents from the crates.io API, which has a rate limit of 1 request/second that we were exceeding. This commit changes the download URL to use the CDN, which has no rate limit. See this post https://www.pietroalbini.org/blog/downloading-crates-io/ for more details.

Test plan

See the CI go green. I also manually tested the change locally to verify it works as expected using the site config

❯ cat ../dev-private/enterprise/dev/external-services-config.json
{
  "RUSTPACKAGES": [
    {
      "dependencies": ["[email protected]"]
    }
  ]
}

Merge request reports