Skip to content

Use streaming in repo-updater to make syncing process more responsive

Created by: mrnugget

Currently the syncing process in repo-updater works in batches: it loads all repositories from each configured external service and then adds/modifies/deletes them from the database.

The drawback of this approach is that if the external service yields thousands and thousands of repositories it takes a long time for a repository to end up in the database and show up in the UI.

That leads to confusion with users thinking that the repo has not been added:

The idea is to change the architecture of repo-updater to stream new and modified repositories. That would make new repos show up immediately after adding a new external service.