Skip to content

Keep repository set in sync with config

Created by: sqs

User story

As a site admin, I configure Sourcegraph with a personal access token. I want to share Sourcegraph with my team, but I don't want Sourcegraph to be configured with all of the repositories that my token has access to. Either I update the token, or I limit the scope of synced repositories (i.e. via a search query or regex) and the repositories that are no longer accessible immediately disappear from my Sourcegraph instance.

Product changes

  • Mirrored repositories will be always kept in sync with associated external service configuration.
  • Mirrored repositories will be configured exclusively via associated external service configuration.
    • Enabled and disabled state will be deprecated and removed from the UI.
    • Explicit deletion of individual repositories will be removed from the UI.

Planned work for 3.3

  • Deployment
    • Instrument Syncer with Prometheus metrics (#3398)
    • Instrument server endpoints with Prometheus metrics (#3398)
    • Instrument new code with tracing. (#3398)
  • Documentation
    • Update GitHub external service documentation.
      • Document new repositoryQuery default. (#2895)
      • Document new exclude setting.
    • Update GitLab external service documentation (#3042 (closed))
    • Update Bitbucket Server external service documentation.
      • Document new repositoryQuery setting.
      • Document new repos setting.
      • Document new exclude setting.
      • Document new required username field.
    • Write blog post about how the new syncer works.
  • User Interface
  • Common
    • Ensure enabled state migration is safe to run many times (#3046)
    • Retry migrations instead of exiting. Verify that Sourcegraph is usable without repo-updater (#3046)
    • Trigger full sync when an external service is deleted. (#3290)
    • Ensure that database is migrated once the frontend is ready. Right now waiting for the frontend doesn't ensure that migrations were ran. We either change the wait for frontend method to wait for migrations too, or make repo-updater also run migrations. (#3286)
    • GraphQL API's setRepositoryEnabled deprecation path (#3162 (closed), #3252)
    • Handle repo renames in the scheduler (#2609 (closed))
    • Default feature flag to on.
    • Use centralized service discovery mechanism to retrieve Postgres connection data. (cc: @slimsag)
  • GitHub
    • Implement GitHubSource and enable new syncer for GitHub external services (#2266)
    • Implement github.exclude to support enabled state deprecation. (#2649)
    • Change default github.repositoryQuery to ["none"] and migrate existing configs (#2785, #2792)
    • Implement enabled state deprecation migration of GitHub external services (#2887)
      • Migrate all disabled repos to github.exclude list.
      • Migrate all enabled repos that aren't yielded by any GitHub source to github.repos list.
      • Deprecate github.initialRepositoryEnablement setting.
    • Show error to admin for repositoryQuery with more than 1000 results (#2562 (closed))
    • repositoryQuery may return incomplete results (#3363 (closed))
    • Gracefully handle non-existent repos in repos list. (#3027)
    • Update UI to remove enabled state for GitHub repos.
    • Update UI to remove explicit deletion for GitHub repos.
  • GitLab
    • Implement GitLabSource and enable new syncer for GitLab external services (#3008)
    • Implement gitlab.projects to support enabled state deprecation. (#3027)
      • Gracefully handle non-existent projects in projects list.
    • Implement gitlab.exclude to support enabled state deprecation. (#3014)
    • Change default gitlab.projectQuery to ["none"] and migrate existing configs. (#3028)
    • Extend enabled state deprecation migration with GitLab external services. (#3029)
    • Update UI to remove enabled state for GitLab repos.
    • Update UI to remove explicit deletion for GitLab repos.

Stretch goals for 3.3

  • Implement dry-run mode in Syncer.Sync to support #2866 (closed)
  • Bitbucket Server
    • Implement BitbucketServerSource and enable new syncer for BitbucketServer external services (#3097)
    • Implement bitbucketserver.repos to support enabled state deprecation. (#3097)
      • Gracefully handle non-existent repos in repos list.
    • Implement bitbucketserver.exclude to support enabled state deprecation. (#3097)
    • Implement bitbucketserver.repositoryQuery, set the default to ["none"] and migrate existing configs to have the previous semantically equivalent default of ["?visibility=private", "?visibility=public"] (#3097)
    • Migrate existing Bitbucket Server repos external_id's from projectName/repoName to repoID to support renames. (#3097)
    • Implement enabled state deprecation migration of BitbucketServer external services (#3097)
      • Migrate all disabled repos to bitbucketserver.exclude list.
      • Migrate all enabled repos that aren't yielded by any BitbucketServer source to bitbucketserver.repos list.
      • Deprecate bitbucketserver.initialRepositoryEnablement setting.
    • Update UI to remove enabled state for BitbucketServer repos.
    • Update UI to remove explicit deletion for BitbucketServer repos.

Backlog

  • Create Grafana dashboard for repo-updater
  • Ensure that sourcegraph.com code paths work with the feature flag on. How does manually adding a repo when directly visiting one interact with the new Syncer?
  • Search and replace everywhere: s/Github/GitHub/g
  • Test: Server HTTP endpoints
  • Test: Update from diff in scheduler
  • GitHub supports query configuration to replace repositoryQuery #2562 (closed)

Customers affected

Too many to name individually, but a highlight: