Repos migration triggers abuse detection rate limit on GitHub Enterprise
Created by: nicksnyder
Reported by https://app.hubspot.com/contacts/2762526/company/561806411 after upgrading to 3.3.1
22:15:40 repo-updater | t=2019-04-24T22:15:40+0000 lvl=eror msg=store.done error="migrate.repos-enabled-state-deprecation.sources.list-repos: 1 error occurred:\n\t* Error getting GitHub repository: nameWithOwner=pascal-carole/datachef: request to http://127.0.0.1:3180/repos/pascal-carole/datachef returned status 403: You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.\n\n"
22:15:40 repo-updater | t=2019-04-24T22:15:40+0000 lvl=eror msg=migrate error="migrate.repos-enabled-state-deprecation.sources.list-repos: 1 error occurred:\n\t* Error getting GitHub repository: nameWithOwner=pascal-carole/datachef: request to http://127.0.0.1:3180/repos/pascal-carole/datachef returned status 403: You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.\n\n" waiting=5s
root@i-0b5d6613c1b911c64:/home/byron_grogan#
2:14:17 repo-updater | t=2019-04-24T22:14:17+0000 lvl=eror msg=migrate error="migrate.repos-enabled-state-deprecation.sources.list-repos: 1 error occurred:\n\t* Error getting GitHub repository: nameWithOwner=philip-snowberger/puppet_control: request to http://127.0.0.1:3180/repos/philip-snowberger/puppet_control returned status 403: You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.\n\n" waiting=5s
n.sources.list-repos: 1 error occurred:\n\t* Error getting GitHub repository: nameWithOwner=philip-snowberger/puppet_control: request to http://127.0.0.1:3180/repos/philip-snowberger/puppet_control returned status 403: You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.\n\n" waiting=5s
22:15:28 repo-updater | t=2019-04-24T22:15:28+0000 lvl=eror msg=store.done error="migrate.repos-enabled-state-deprecation.sources.list-repos: 1 error occurred:\n\t* Error getting GitHub repository: nameWithOwner=pascal-carole/datachef: request to http://127.0.0.1:3180/repos/pascal-carole/datachef returned status 403: You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.\n\n"Customer's GitHub Enterprise config looks like this:
{
"gitURLType": "ssh",
"repositoryPathPattern": "{nameWithOwner}",
"token": "<SNIPPED>",
"url": "https://<SNIPPED>",
"repositoryQuery": [
"org:company created:<2016-01-01",
"org:company created:2016-01-01..2017-12-31",
"org:company created:2018-01-01..2018-12-31",
"org:company created:>2019-01-01",
"org:foobarbaz",
"org:third-party"
]
}Does the migration honor all rate limits?
I wonder if this line of code might be the issue: https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/cmd/repo-updater/repos/github.go#L420:2
We are concurrently querying repository queries which might trigger abuse detection mechanisms, even if we are staying under the rate limit. Perhaps we should do this serially.