Skip to content

monitoring: steps for handling "we are working on this alert"

Created by: bobheadxi

For example, https://github.com/sourcegraph/sourcegraph/issues/12816 tracks issues with syntect_server that causes alerts to fire frequently. We should have a process for tracking work on an alert, either through an issue template or handbook process, along the lines of:

  • acknowledge alert fires frequently and silence the alert on Cloud (if relevant)
  • create an issue tracking the alert
  • update PossibleSolutions of the alert to say "go to this issue to chime in with your details, or subscribe to it for updates", which should track work on followups (removing the PossibleSolutions entry, unsilencing the alert, etc.)