Lockfile-indexing-based dependency search (backend)

Created by: mrnugget

• Build out lockfile indexing for Yarn
  • First Yarn (because we can everything from yarn.lock,)
  • Build parser for yarn.lock that creates a full tree https://github.com/sourcegraph/sourcegraph/pull/37544
  • Persist tree to database (not a flat list) (fixed in https://github.com/sourcegraph/sourcegraph/pull/37543)
  • Save metadata about indexing operation ("indexed at commit f00 at 12:45, wth full graph")
• Policy support for lockfile indexing (i.e. "index these repos at HEAD every 15min") (fixed in https://github.com/sourcegraph/sourcegraph/pull/37543)
• Ensure correct data is saved for other languages (fixed in https://github.com/sourcegraph/sourcegraph/pull/37543)
  • For Python/Go: if we can't reconstruct full tree, only persist and return direct dependencies
• Allow querying of dependency tree on service layer (i.e. not via search interface yet)
  • transitive:yes|no|only
  • level 1/2/3
• Retention policy support for lockfile indexing (i.e. "delete these lockfile graphs after 3 weeks")
• Create an admin page/repo page that gives overview of lockfile indexing status
• Write docs on how to start lockfile indexing (fixed in https://github.com/sourcegraph/sourcegraph/pull/37543)
• Turn off lazy dependency search and only use previously-persisted results from lockfile indexing (alert user?) (fixed in https://github.com/sourcegraph/sourcegraph/pull/37543)
• Clean up dependenciesSource duplication between internal/repos/depdencies.go and cmd/gitserver/server/vcs_dependencies_syncer.go
• Handle duplicate dependencies: https://github.com/sourcegraph/sourcegraph/pull/37543#discussion_r916823644

Possibly

• Also parse package.json in combination with yarn.lock to determine direct dependencies when finding cyclic graph. See https://github.com/sourcegraph/sourcegraph/pull/37544/files#r917141807

Notes

Olaf's idea for creating full tree/graph for Maven/pom.xml:

Idea for Java dependencies search: support only pom.xml files and implement a lockfile parser that shells out to mvn dependency:tree to get the full dependency graph. There are options to emit structured XML files or graphviz dot files. Maven is a declarative build tool so it should be safe to invoke this command unlike the equivalent functionality for Gradle, which requires arbitrary code execution https://gist.github.com/olafurpg/d434a4cdefd7e92e30d2d1876cd48efb