Something went wrong while fetching comments. Please try again.
Created by: ryanslade
This only affects sourcegraph.com
We're seeing an increase in failed background syncs and looking into the logs we can see that most of them are due to our internal rate limiter being exceeded.
Since we usually wait on the limiter, the reason it is failing is that the amount of time we'd need to wait would exceed the time on the context we're using in the background sync, which was set to one minute. In this case, it bails out immediately with an error since the context would expire before the rate limiter would allow the request.
I've increase the context on our background context to three minutes which should decrease these failures. One potential downside is we're going to have more running goroutines but they're pretty cheap and so it should be ok.
I also updated a few places in our code where I noticed we were increasing metrics using Add(1) instead of Inc()