Sync single external service
Created by: ryanslade
This PR makes two main changes:
- We sync each external service independently
- The syncing process now happens using the
workerutil
package
Syncer.Run
now enqueues external services into a job queue that are due to be synced every minute.
These jobs are picked by up to three sync workers concurrently each worker syncs one service using the new Syncer.SyncExternalService
method.
We rely on database triggers to mark repos as deleted once they no longer exist in any external service. This is based on the state of the external_service_repos
table which tracks the relationship between repos and external services.
Triggers also exist that remove rows from external_service_repos
when a repo or external service is deleted or soft deleted.
@keegancsmith We'd love your feedback specifically on the the code related to SyncSubset
as we are not 100% sure that this still works as expected.