Skip to content

authz/github: user-centric perms sync from team/org perms caches

Warren Gifford requested to merge team-permissions-sync into main

Created by: bobheadxi

Implements an approach to the Teams and Organizations syncing outlined in Scaling repository permissions syncing. Currently, rate limiting means that for the benchmark deployment size of 10k repos / 5k members / 100 teams / 1 org, sync times today takes ~4 days.

This approach adds team/organization (also referred to as groups) permission sync behind the Provider abstraction. More specifically, we extend the GitHub provider's FetchUserPermsByToken to use cached groups (orgs or teams) permissions. Then we can reduce API calls by:

  • narrowing the scope of listing user repos (just list affiliation: owner,collaborator instead of everything)
  • reduce the amount of repo perms sync'd multiple times by having org and team permissions relevant to a user be pulled from a per-Provider cache

This approach should be much better for a "best-case" scenario where no users have owner,collaborator permissions, with sync time described as:

hours = (users + (teams + orgs) * repositories/100) / 5000

Ths means that for the benchmark deployment size:

  • sync times will take ~3 hours in the best-case scenario
  • sync times will still take ~4 days in the worst-case scenario (where all permissions are distributed by giving users direct collaborator access)

This PR also introduces options for performing permission syncs that can be provided at various levels of permissions syncing. For now, this is only added to user permissions, and is primarily used to provide a InvalidateCaches flag that signals the Provider to invalidate caches encountered during the sync, which is currently only used by webhook handlers to invalidate caches when a team/org-related event is received.

Note that as of this PR, the above only applies to user permissions sync - an implementation for repos will follow.

TODOs

Related PRs

  • #24311
  • #24316
  • #24318
  • #24326
  • #24328
  • #24329
  • #24332
  • #24344

Merge request reports