Skip to content

monitoring: adjust thresholds for noisy critical alerts

Warren Gifford requested to merge distribution/critical-alerts-tweaks into master

Created by: bobheadxi

Adjust thresholds for frequently firing critical alerts, as seen on sourcegraph.com:

  • gitserver: disk_space_remaining
  • precise-code-intel-bundle-manager: disk_space_remaining

Dogfooding alerting (https://github.com/sourcegraph/sourcegraph/issues/5370) will probably involve subscribing our opsgenie to all critical alerts, so I'm thinking now might be a good time to assess whether or not these are actually critical. From our docs:

Critical alerts are guaranteed to be a real issue with Sourcegraph.

I'm assuming this means that in the long term, customers (and us) should be feel safe setting up notifications for Sourcegraph without being worried about being paged for things that can safely be ignored.

Slack thread: https://sourcegraph.slack.com/archives/CJX299FGE/p1594113545255400

Merge request reports