Skip to content

repo-updater: Stream inserts

Warren Gifford requested to merge core/stream-add into master

Created by: keegancsmith

Syncing requires fetching from all sources before it can update the database. For large instances this leads to a long time (minutes) from the time an admin makes a change to an external service to seeing the effects. The most common complaint is adding repositories to the external service, and not seeing them. This commit will eagerly add new repositories into the store before running the full sync. This is a much smaller change than making the sync engine fully stream based.

This is an alternative approach to making repo-updater completely stream based. The important observation is that users complain that they add a repository to an external service and don't see it. This change doesn't adjust how repo-updater's core sync algorithm works. Instead it will eagerly insert any repository from a source which is not already in the store.

Alternative to https://github.com/sourcegraph/sourcegraph/pull/5526

Fixes https://github.com/sourcegraph/sourcegraph/issues/5145

Merge request reports

Loading