search: dependencies search prototype
Created by: tsenart
Context
This PR introduces dependency search of package repos as a prototype of RFC 593.
It introduces a new repo:dependencies
predicate (and its alias deps
) that receive a repo-regex@revA:revB:...
argument which will be an input to resolving the set of repos to search over. We handle it by finding all matching repos, fetching supported lockfiles from gitserver, parsing them and finally converting those dependencies to repos to search over, triggering a sync of all existing package host external services at the end.
This means the first time we use r:deps
with repos that contain new dependencies in the instance, we won't be able to search those immediately, but they should be shortly available thereafter (i.e. lazy syncing).
We only support searching over NPM package repos, since we only have a package-lock.json
parser implemented. Once more parsers are implemented (e.g. yarn.lock) for package hosts we already support, we will expand the set of searchable dependency repos.
For now we are not storing the relationships in a way that we can query directly from the database — every single search with dependencies
in it needs to re-parse lockfiles.
Test plan
Unit, integration and manual end to end tests.