monitoring: add observability.silenceAlerts

Created by: bobheadxi

Adds observability.silenceAlerts, which deploys silences to sourcegraph/prometheus's built-in Alertmanager.

updated

{
  "observability.silenceAlerts": [
    "warning_gitserver_disk_space_remaining",
    "critical_gitserver_command_duration_test"
  ]
}

Closes #11210 (closed)

Considerations

• Specificity: I'm opting to disallow broad silences (ie all of name, level, service are required and a user cannot mute all "service": "gitserver" alerts). Regex is possible, but not allowed at the moment. Open to thoughts on this though

• Implementation detail: Alertmanager requires a start and end date for each alert. Right now, I've set this to 10 years... should be fine right?

• Via a normal user flow of:

  1. see site banner for critical alert
  2. go to grafana

  there does not seem to be a clear way to get the name of an alert at the moment - not sure if this is within the scope of this PR to address, and if there's a good way to go about it (maybe replace the human-readable description in the main table with alert name?) => resolved via generating full alert names in docs