git: implement protocol v2 support in ls-remote
Created by: keegancsmith
We currently rely on ls-remote in a few key places in our git infrastructure. We can significantly speed it up by using git's protocol v2. To do this would require adding support in upstream git (so get out your C editor). See this mail thread about the problem and what I proposed. https://public-inbox.org/git/[email protected]/T/#u
How to contribute to git https://mirrors.edge.kernel.org/pub/software/scm/git/docs/MyFirstContribution.html
Some debug details on perf wins:
We do a lot more work in ls-remote than fetch
GIT_TRACE_PERFORMANCE=1 GIT_TRACE_PACKET=1 git ls-remote origin HEAD
GIT_TRACE_PERFORMANCE=1 GIT_TRACE_PACKET=1 git fetchls-remote
11:51:22.159766 pkt-line.c:80 packet: git< # service=git-upload-pack
11:51:22.160294 pkt-line.c:80 packet: git< 0000
11:51:22.160307 pkt-line.c:80 packet: git< version 2
11:51:22.160319 pkt-line.c:80 packet: git< agent=git/github-g6c540c87b5b7
11:51:22.160328 pkt-line.c:80 packet: git< ls-refs
11:51:22.160333 pkt-line.c:80 packet: git< fetch=shallow filter
11:51:22.160338 pkt-line.c:80 packet: git< server-option
11:51:22.160344 pkt-line.c:80 packet: git< 0000
11:51:22.160464 pkt-line.c:80 packet: git< version 2
11:51:22.161382 pkt-line.c:80 packet: git< agent=git/github-g6c540c87b5b7
11:51:22.161392 pkt-line.c:80 packet: git< ls-refs
11:51:22.161399 pkt-line.c:80 packet: git< fetch=shallow filter
11:51:22.161406 pkt-line.c:80 packet: git< server-option
11:51:22.161411 pkt-line.c:80 packet: git< 0000
11:51:22.161418 pkt-line.c:80 packet: git> command=ls-refs
11:51:22.161432 pkt-line.c:80 packet: git> agent=git/2.26.0
11:51:22.161438 pkt-line.c:80 packet: git> 0001
11:51:22.161444 pkt-line.c:80 packet: git> peel
11:51:22.161449 pkt-line.c:80 packet: git> symrefs
11:51:22.161454 pkt-line.c:80 packet: git> 0000
11:51:22.161487 pkt-line.c:80 packet: git< command=ls-refs
11:51:22.161509 pkt-line.c:80 packet: git< agent=git/2.26.0
11:51:22.161516 pkt-line.c:80 packet: git< 0001
11:51:22.161523 pkt-line.c:80 packet: git< peel
11:51:22.161529 pkt-line.c:80 packet: git< symrefs
11:51:22.161534 pkt-line.c:80 packet: git< 0000
11:51:22.809304 pkt-line.c:80 packet: git< a7d14a44285d3ec4b25bf9e3b7df701221350661 HEAD symref-target:refs/heads/master
11:53:42.917009 pkt-line.c:80 packet: fetch< version 2
11:53:42.918431 pkt-line.c:80 packet: fetch< agent=git/github-g6c540c87b5b7
11:53:42.918446 pkt-line.c:80 packet: fetch< ls-refs
11:53:42.918456 pkt-line.c:80 packet: fetch< fetch=shallow filter
11:53:42.918466 pkt-line.c:80 packet: fetch< server-option
11:53:42.918475 pkt-line.c:80 packet: fetch< 0000
11:53:42.918484 pkt-line.c:80 packet: fetch> command=ls-refs
11:53:42.918507 pkt-line.c:80 packet: fetch> agent=git/2.26.0
11:53:42.918516 pkt-line.c:80 packet: fetch> 0001
11:53:42.918524 pkt-line.c:80 packet: fetch> peel
11:53:42.918532 pkt-line.c:80 packet: fetch> symrefs
11:53:42.918542 pkt-line.c:80 packet: fetch> ref-prefix refs/heads/
11:53:42.918551 pkt-line.c:80 packet: fetch> ref-prefix refs/tags/
11:53:42.918562 pkt-line.c:80 packet: fetch> 0000One line patch to
git on master [!?] took 2s
❯ PATH="$PWD:$PATH" GIT_TRACE_PACKET=1 git -c protocol.version=2 ls-remote --symref origin HEAD
13:06:59.906330 pkt-line.c:80 packet: git< # service=git-upload-pack
13:06:59.907029 pkt-line.c:80 packet: git< 0000
13:06:59.907048 pkt-line.c:80 packet: git< version 2
13:06:59.907061 pkt-line.c:80 packet: git< agent=git/github-g6c540c87b5b7
13:06:59.907069 pkt-line.c:80 packet: git< ls-refs
13:06:59.907076 pkt-line.c:80 packet: git< fetch=shallow filter
13:06:59.907083 pkt-line.c:80 packet: git< server-option
13:06:59.907090 pkt-line.c:80 packet: git< 0000
13:06:59.907249 pkt-line.c:80 packet: git< version 2
13:06:59.908564 pkt-line.c:80 packet: git< agent=git/github-g6c540c87b5b7
13:06:59.908584 pkt-line.c:80 packet: git< ls-refs
13:06:59.908600 pkt-line.c:80 packet: git< fetch=shallow filter
13:06:59.908614 pkt-line.c:80 packet: git< server-option
13:06:59.908626 pkt-line.c:80 packet: git< 0000
13:06:59.908638 pkt-line.c:80 packet: git> command=ls-refs
13:06:59.908669 pkt-line.c:80 packet: git> agent=git/2.26.0.51.ga7d14a4428.dirty
13:06:59.908684 pkt-line.c:80 packet: git> 0001
13:06:59.908697 pkt-line.c:80 packet: git> peel
13:06:59.908709 pkt-line.c:80 packet: git> symrefs
13:06:59.908712 pkt-line.c:80 packet: git< command=ls-refs
13:06:59.908723 pkt-line.c:80 packet: git> ref-prefix HEAD
13:06:59.908736 pkt-line.c:80 packet: git> 0000
13:06:59.908735 pkt-line.c:80 packet: git< agent=git/2.26.0.51.ga7d14a4428.dirty
13:06:59.908750 pkt-line.c:80 packet: git< 0001
13:06:59.908765 pkt-line.c:80 packet: git< peel
13:06:59.908888 pkt-line.c:80 packet: git< symrefs
13:06:59.908902 pkt-line.c:80 packet: git< ref-prefix HEAD
13:06:59.908907 pkt-line.c:80 packet: git< 0000
13:07:00.475167 pkt-line.c:80 packet: git< a7d14a44285d3ec4b25bf9e3b7df701221350661 HEAD symref-target:refs/heads/master
13:07:00.790427 pkt-line.c:80 packet: git< 0000
ref: refs/heads/master HEAD
a7d14a44285d3ec4b25bf9e3b7df701221350661 HEAD
git on master [!?] took 5s
❯ git diff
diff --git a/builtin/ls-remote.c b/builtin/ls-remote.c
index 6ef519514b..12d3af177a 100644
--- a/builtin/ls-remote.c
+++ b/builtin/ls-remote.c
@@ -91,6 +91,8 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
}
}
+ argv_array_push(&ref_prefixes, "HEAD");
+
if (flags & REF_TAGS)
argv_array_push(&ref_prefixes, "refs/tags/");
if (flags & REF_HEADS)