Skip to content

repo-updater: Sync single external service at a time using workerutil

Warren Gifford requested to merge sync_single_external_service into main

Created by: ryanslade

This PR makes two main changes:

We sync each external service independently The syncing process now happens using the workerutil package Syncer.Run now enqueues external services into a job queue that are due to be synced every minute.

These jobs are picked by up to three sync workers concurrently each worker syncs one service using the new Syncer.SyncExternalService method.

We rely on database triggers to mark repos as deleted once they no longer exist in any external service. This is based on the state of the external_service_repos table which tracks the relationship between repos and external services.

Triggers also exist that remove rows from external_service_repos when a repo or external service is deleted or soft deleted.

We've testing syncing up to 500k repos successfully.

Merge request reports