GitHub GraphQL ratelimit is not respected

Created by: eseliger

The calls to the GitHub GraphQL API in our GitHub client code don't respect the rate limit imposed on us by them.

Problems this causes:

• The reconciler can run into rate limiting issues
• Listing repos can fail
• Syncing changesets in the background can fail (but we do have a self-imposed rate limit here AFAIK)

What we observed so far:

• We do have two rate limiters in the client, one self imposed rateLimiter, one based on GitHubs rate limit rateLimitMonitor
• The rateLimiter tracks GraphQL and REST calls in the same bucket, although they likely have separate rate limits (to be confirmed)
• The rateLimitMonitor is updated by passing in the response headers, but GraphQL doesn't return those, but exposes the fields through the response
• Querying the rateLimit property through GraphQL seems to be cost neutral, against the docs which state it always at least costs 1 token
• The rateLimiter is global to repo-updater and hence being used in the reconciler
• rateLimitMonitor is not shared globally
• Reconciler fails with rate limit errors, making the changeset end up in the errored state, but it will be retried eventually.

Ideas for a solution:

• Make the rateLimitMonitor also update from GraphQL results
• Use the rateLimitMonitor in the requestGraphQL method on the client, not only explicitly at certain places
• Add a function on the monitor to take the suggested rate limit out of the values reported by the monitor PLUS the one of our internal rate limiter
• Make the rateLimitMonitor globally shared, so new clients still know about the previous requests limits

Future improvements:

• Merge the monitor and internal rate limiter into a single structure that can account for the combined limitation of both (and split across GraphQL and REST rate limits)
• Do proper timeouts and timeout error handling on the reconciler so these become enqueued back without erroring

cc @mrnugget @ryanslade