Configuration refactoring + management console

Created by: slimsag

Overview

• This change refactors our configuration code (in a 100% backwards compatible way) to be read from the database instead of from a config file.
• Existing deployments that are still configured to use a config file (e.g. via SOURCEGRAPH_CONFIG_FILE) will automatically be migrated: That config file will be split and loaded into the DB. After which, the config file specified via SOURCEGRAPH_CONFIG_FILE will never be used again and can be removed.
• The new management console service added in this PR allows modifying the critical Sourcegraph configuration options. Other options are still edited through the site configuration in the site admin area.

Depends on:

• db: rename "site config" to "global state" #1083
• mv cmd/frontend/db/dbconn to pkg/dbconn #1084
• pkg/conf/confdb: new package for storing critical + site configuration in DB #1089
• add stub cmd/management-console implementation #1109
• make more conf.Watch usages async #1148

Merge checklist:

• Factor out the change to rename the db.SiteConfig struct into a separate PR and land it.
• Factor out the pkg/dbconn change into a separate PR.
• Factor out DB change into a separate PR.
• Factor out stub management-console binary into a separate PR.
• Factor out / finish writing async conf.Watch changes into a separate PR.
• Factor out confdb.SetDefault* call change into a separate PR. cannot
• Figure out how to deploy this to production and dogfood. (What query populates the DB? How is it run? etc.) added auto migration code
• Remove SOURCEGRAPH_CONFIG_FILE from deployment configs. using for migration
• Implement conf.Edit
• Clarify defaultDevAndTestingConfiguration applies to all tests and conf.Mock should be used for specific test configuration.
• Implement validation of default configs.
• Implement config writes.
• Figure out how we import our private dev config.
• Write management console UI
• Add saving support to management console UI
• Add schema to Monaco
• Add private dev config override in dev/start.sh.
• When Sourcegraph Server frontend dies, should the container exit or should the management console still be accessible?
• Add basic auth requirement
• Figure out how to silence ⚠️ Could not load existing sourcemap warnings?
• Prompt site admins with plaintext password + dismiss button in regular app.
• Split site config in dev-private repo into critical-config.json and site-config.json: https://github.com/sourcegraph/dev-private/pull/1
• Add proper config defaults for each deployment type.
• Fix compilation issues with the webapp code
• Fix CI / any broken tests.
• Bug: management console imports pkg/conf and thinks it is a client, thus pulling config from frontend. Not good.
• Check docs are backwards compat (if you expose a port that is not listening via docker run --expose, what happens?) -> nothing.
• Send PR to run management-console in deploy-sourcegraph[-docker] https://github.com/sourcegraph/deploy-sourcegraph/pull/151

Future (required for 3.0 preview):

• (3.0-preview) Update CHANGELOG to reflect this change accurately.
• (3.0-preview) Update all docs to reference the management console. Clearly document how migration occurs, and what changes users should expect (needing to visit the management console). Add management-console documentation. Document how to upgrade old instances (new port exposure needed). https://docs.sourcegraph.com/admin/management_console
• (3.0-preview) Update all docs to reference the management console.

Future (ideally in 3.0 preview):

• A bad ExternalURL should not kill the frontend process. It should just make it "do nothing".
• Indicate through mgmt console when license is expired (locked out of frontend in that case)
• Update deploy-sourcegraph-docker
• Update deploy-sourcegraph-docker's config.json to match appURL -> externalURL change in https://github.com/sourcegraph/deploy-sourcegraph/pull/146

Future: 3.0 official

• (3.0) For now, validation errors for critical config are only displayed in the same area we display them for the site config. This is unfortunate and will be filed as a bug.
• (3.0) Pass GitServers list via conf.Get().ServiceConnections.GitServers
• (3.0) Remove siteConfigurationResolver.Source
• (3.0) // TODO(slimsag): future comments.
• (3.0) Add config usage doc:
  • Do not rely on conf.Get or conf.Watch in a synchronous way.
  • Bad configuration should NEVER take down the frontend. It is assumed if the frontend goes down, it is because of a bug not because of bad configuration. In the worst case, bad configuration should just cause the frontend to do nothing until configuration is fixed.
  • Missing configuration should always lead to stricter security: e.g. if no authentication is provided, Sourcegraph should become 100% locked down.

This PR does not need to update the CHANGELOG because it will be done subsequently after proper documentation is written and can be linked to.

Notes about documentation

The following snippets are things that I will update our docs to include in the near future:

TODO: create this page: https://docs.sourcegraph.com/admin/management_console

TODO: add to docs:

Resetting the management console password manually

1. Install Python
2. pip install bcrypt
3. Encrypt your password:

PASSWORD='abc123' python -c "import bcrypt; import os; print(bcrypt.hashpw(os.environ['PASSWORD'], bcrypt.gensalt(15)))"

4. Open a psql prompt on your Sourcegraph instance and run:

UPDATE global_state SET mgmt_password_bcrypt='my-encrypted-password';

TODO: add to docs:

The following site config options have been moved to the management console:

auth.providers
auth.public
auth.sessionExpiry
auth.userOrgMap
externalURL
htmlBodyBottom
htmlBodyTop
htmlHeadBottom
htmlHeadTop
httpStrictTransportSecurity
httpToHttpsRedirect
licenseKey
lightstepAccessToken
lightstepProject
log
tls.letsencrypt
tlsCert
tlsKey
update.channel
useJaeger

TODO: Update deploy-sourcegraph docs, see as reference: https://github.com/sourcegraph/deploy-sourcegraph/pull/140/files