Skip to content

repo-updater: Improve performance when calculating notClonedCount

Administrator requested to merge core/perf-clone-status into master

Created by: mrnugget

This contains the first possible (to be verified!) performance improvement outlined in this comment. Namely option (a) in 17a5b70 and then option (b) in 24f7d1ebcd

Test plan: Deploy to k8s.sgdev.org and compare performance

Benchmarks

This is the vegeta command I used on my local machine to query the status messages once every second. Since the status message cache has a TTL of 30 seconds, I let it run for >2min so that I can measure the max latency when the cache is busted.

jq -ncM '{method: "POST", url: "https://k8s.sgdev.org/.api/graphql", body: {"query":"query StatusMessages {\n                statusMessages {\n                    message\n                    type\n                }\n            }\n        "} | @base64, header: {"Authorization": ["token <TOKEN>"]}}' | vegeta attack -format=json -rate=1/1s | vegeta report -every 1s

master — Vegeta report

Requests      [total, rate]            131, 1.01
Duration      [total, attack, wait]    2m10.206830444s, 2m9.996781545s, 210.048899ms
Latencies     [mean, 50, 95, 99, max]  750.229049ms, 204.037912ms, 4.475514686s, 6.102255507s, 6.915630101s
Bytes In      [total, mean]            13100, 100.00
Bytes Out     [total, mean]            21877, 167.00
Success       [ratio]                  100.00%
Status Codes  [code:count]             200:131
Error Set:

Note the max latency of 6.9s, that's the important number we want to get down.

17a5b7000088d9f775808ea11ef0bac411419e5a - Vegeta report

This is option (a): using a COUNT in repo-updater and getting a list of cloned repos from gitserver.

Requests      [total, rate]            258, 1.00
Duration      [total, attack, wait]    4m17.126976822s, 4m17.001316592s, 125.66023ms
Latencies     [mean, 50, 95, 99, max]  242.134555ms, 202.935664ms, 582.110812ms, 1.302482009s, 1.804293883s
Bytes In      [total, mean]            25800, 100.00
Bytes Out     [total, mean]            43086, 167.00
Success       [ratio]                  100.00%
Status Codes  [code:count]             200:258
Error Set:

Max latency is down to 1.8s!

24f7d1ebcd - Vegeta report

This is option (b): using a COUNT in repo-updater and getting a count of cloned repos from gitserver.

Requests      [total, rate]            451, 1.00
Duration      [total, attack, wait]    7m30.140661628s, 7m30.007242931s, 133.418697ms
Latencies     [mean, 50, 95, 99, max]  183.504228ms, 142.252561ms, 457.949139ms, 958.433507ms, 1.277211677s
Bytes In      [total, mean]            45100, 100.00
Bytes Out     [total, mean]            75317, 167.00
Success       [ratio]                  100.00%
Status Codes  [code:count]             200:451
Error Set:

Max latency is down to 1.2s!

Merge request reports