Skip to content

executor: Implement shallow clones and sparse checkouts

Warren Gifford requested to merge es/clone-opts-executor into main

Created by: eseliger

To improve performance, we now offer these two new clone modes. They will be used in a follow-up PR by SSBC to replace repo zip archive fetching with proper clones that are faster because git can stream, unlike zip.

This also adds a flag to change how local workspaces are set up: the repo can now be cloned into a subdirectory so the script and other additional files don't clutter the repo content.

Does the new way we use git break auto indexing in any way I can't predict? It works properly locally. Also, I think I heard you say that you need full clones for LSIF so code intel doesn't benefit too much from this yet, but maybe there's potential in the future to use shallow clones and sparse checkouts too? Or we can find a way to get it to work without the entire repo; this brings down network traffic for sg/sg from ~800MB to ~52MB, one of the big limitations on our executors in prod today (network and disk IO).

Test plan

Manually checked SSBC and auto-indexing still seem to work. The two new modes aren't used yet in this PR, so this will mostly be an invisible change until we make use of it in a follow-up PR.

Merge request reports

Loading