monitoring: adjust thresholds for noisy critical alerts
Created by: bobheadxi
Adjust thresholds for frequently firing critical
alerts, as seen on sourcegraph.com:
- gitserver: disk_space_remaining
- precise-code-intel-bundle-manager: disk_space_remaining
Dogfooding alerting (https://github.com/sourcegraph/sourcegraph/issues/5370) will probably involve subscribing our opsgenie to all critical alerts, so I'm thinking now might be a good time to assess whether or not these are actually critical. From our docs:
Critical alerts are guaranteed to be a real issue with Sourcegraph.
I'm assuming this means that in the long term, customers (and us) should be feel safe setting up notifications for Sourcegraph without being worried about being paged for things that can safely be ignored.
Slack thread: https://sourcegraph.slack.com/archives/CJX299FGE/p1594113545255400
Merge request reports
Activity
Created by: codecov[bot]
Codecov Report
Merging #11988 into master will not change coverage. The diff coverage is
n/a
.@@ Coverage Diff @@ ## master #11988 +/- ## ======================================= Coverage 50.04% 50.04% ======================================= Files 1515 1515 Lines 88442 88442 Branches 6770 6770 ======================================= Hits 44262 44262 Misses 40240 40240 Partials 3940 3940
Flag Coverage Δ #go 54.49% <0.00%> (ø)
#storybook 10.74% <0.00%> (ø)
#typescript 36.58% <0.00%> (ø)
#unit 49.66% <0.00%> (ø)
Created by: bobheadxi
Merging this for now and opening https://gitlab.sgdev.org/root/sourcegraph/-/issues/12011 to track other alerts we might want to adjust!