packages: don't abort syncing process if one package fails
Created by: mrnugget
We ran into this after discovering that an invalid name inserted into
the lsif_dependency_repos
table could cause the whole syncing process
to stop.
What would happen:
- Closure does
return err
- The
errgroup.WithContext
would cancel thectx
since one goroutine returned an error - All already running goroutines would finish (maximum 32 - size of semaphore)
- Next call to
ListDependencyRepos
would fail, because the context is cancelled. - Whole loop is stopped and we end up with two errors: the one actual
error from the invalid package and the "context canceled" from
ListDependencyRepos
.
This changes the behaviour to make use of the infrastructure and report
an error only via SourceResult
. That causes the error to be recorded
and the sync job to be marked as failed, but it still syncs the other
packages.
Test plan
- Tested this manually by adding an NPM package host with multiple valid packages to my instance, then doing an
insert into lsif_dependency_repos (name, version, scheme) values ('@automapper/classes/transformer-plugin', '0.0.0', 'npm');
to add the invalid package from production. That way I could check what happens and how my change here fixes it.