Repo-updater component always outputs debug logs
Created by: christinelovett
https://app.hubspot.com/contacts/2762526/company/1712883009 https://sourcegraph.atlassian.net/jira/servicedesk/projects/SG/queues/custom/1/SG-313
We deploy sourcegraph to kubernetes, similar to https://github.com/sourcegraph/deploy-sourcegraph One component of this deployment, is repo-updater https://github.com/sourcegraph/sourcegraph/tree/master/cmd/repo-updater
This depends on a log library log15 https://github.com/inconshreveable/log15 which:
- Logs all levels by default
- Logs non-info levels to stderr
This means that repo-updater always writes its debug-level logs to stderr A side effect of this is that when sending logs from our k8s deployment to stackdriver, all the debug messages are logged with 'ERROR' severity, since they appear on stderr and stackdriver has no other way to get insight into the severity. This causes our logging dashboards to show 100,000s of errors each day.
There are two solutions that would work really well for us either seperately or together, with the first being the easiest to implement:
- Allow configuring log level for repo-updater via a flag (so we can suppress debug output in our deployment, and generally save on logging expenses)
- Use structured (json) logging, with a valid severity field (which stackdriver and other logging systems can parse) (https://cloud.google.com/run/docs/logging#special-fields)