Skip to content

repo-updater: Deterministic pagination from github and gitlab API

Warren Gifford requested to merge core/gh-deterministic-api-order into master

Created by: keegancsmith

Previously we did this so we would add recently pushed repositories first. However, with paged based pagination if a repository is pushed to while we are querying we may miss it. This is the likely cause of repositories occasionally being deleted at a customer.

Instead we use the default sort order. In the case of GitLab this is by created_at which is perfect since that will have a consistent order. For GitHub it is "best match" which assigns a score based on the query. This could not be deterministic. However, I would be surprised if it wasn't. There doesn't seem to be an option you can pass in to GitHub which is based on an immutable value.

Updated testdata with the following commands:

ruplacer --go sort=pushed\& ''
ruplacer --go order_by=last_activity_at\& ''

Merge request reports

Loading