Skip to content

Tracking: Soft launch of indexed search over user added public code

Created by: ryanslade

Database changes

  • Create the external_service_repos table and migrate external_service sources column data to that table. Update stores to read sources data from that new table. #12789
  • Add namespace_user_id column to the external_services table and update stores to load and store data from that column. #12701 (closed)

Security

Repo syncing

Fetching external services

Search

  • Amend search logic to fetch a list of repos configured by a user and include them by default when no repo filter is provided. Ensure that repos added by users are included in the search index.

Handling deletion

  • When a user is deleted (both soft and hard), update their external services deleted_at column. #12965
  • Ensure any entry in external_service_repos is deleted in cascade
  • When an external service is (soft or hard?) deleted, determine which repos are orphaned by counting the number of references in the external_service_repos table and marked them as deleted. This must be done within the same transaction as the one deleting the external service. We need to be careful that if a repo is SOFT deleted it can be added later without causing this index to fail: "repo_external_unique_idx" UNIQUE, btree (external_service_type, external_service_id, external_id) #13273

Progressive rollout

  • Add the external_service_user_mode feature flag as part of the configuration. By default, it will be disabled but if it contains public it will be considered enabled for public code. Anything other than public will disable the flag. Use the flag to protect the external service config page and related GraphQL api to users. #13052
  • Add external_service_user_allow_list configuration option. It will contain a list of regex patterns that must be used to limit the users who will be able to create external services if the external_service_user_mode flag is enabled. https://github.com/sourcegraph/sourcegraph/pull/13099
  • Add external_service_user_allow_percentage configuration option. It will contain a value from 0 to 100. If the external_service_user_mode flag is enabled, use that value to limit the number of users that can create external services, by hashing their user id and applying a modulo 100 on it. If the result is under that value, the user has access to that feature. https://github.com/sourcegraph/sourcegraph/pull/13099
  • If a user is already allowed by external_service_user_allow_list, ignore this check https://github.com/sourcegraph/sourcegraph/pull/13099

NOTE: The above three items can all be solved by checking for tag existing on a user. For now, if a user has the AllowUserExternalServicePublic tag they will be allowed to add their own external services.

  • Create the SQL script that opts in a certain percentage of users. It should be based on a hash of the user id so that running it again with an increased percentage has the desired effect.

This seems to do the trick:

-- Get a repeatable sample of 10%
WITH sample as (SELECT id FROM users TABLESAMPLE BERNOULLI(10) REPEATABLE(1))
-- Update users in the sample
UPDATE users SET tags = array_append(tags, 'AllowUserExternalServicePublic')
WHERE EXISTS (SELECT id from sample)
-- Unless they are already tagged
AND NOT(tags @> ARRAY['AllowUserExternalServicePublic']);

Metrics

  • Track number of code host connections added by users #13589
  • Track total number of repositories added by users #13589
  • Average repos per user can be derived from the above #13589
  • Track whether a user has added any external service (@ebrodymoore will help with this)
    • Given the above, we can derive the visits per user with their own external services vs without
  • Track when feature flag is enabled (@unknwon: I think the metrics can be exposed regardless)
  • Syncer metrics need to include external service id https://github.com/sourcegraph/sourcegraph/pull/13742

Abuse mitigation

Load testing

  • TODO

UI Changes

  • Replicate pages for “manage repositories”. #13095
  • Backend changes to allow CRUD of external services by users. #13173
  • Web app should respect feature flag in JSContext per user. #13095