monitoring: only send specified alerts

Created by: michaellzc

close https://github.com/sourcegraph/sourcegraph/issues/36320

This PR addes a new config attribute in observability.alert, named alerts, where user can configure to only receive alerts from the specific list

Test plan

sg run redis-postgres

sg start monitoring-alerts

Configure site-config

below alerts will always be there on a local environment

  "observability.alerts": [
    {
      "level": "critical",
      "alerts": [
        "warning_frontend_hard_error_search_responses",
        "warning_worker_worker_job_codeintel-janitor_count",
        "critical_worker_worker_job_codeintel-janitor_count"
      ],
      "notifier": {
        "type": "slack",
        "url": "Bring Your Own Webhook (BYOW)"
      }
    }
  ]

watch your slack webhook destination

you can also try a different combination

  "observability.alerts": [
    {
      "level": "critical",
      "alerts": [
        "warning_frontend_hard_error_search_responses",
        "warning_worker_worker_job_codeintel-janitor_count",
        "critical_worker_worker_job_codeintel-janitor_count"
      ],
      "notifier": {
        "type": "slack",
        "url": "Bring Your Own Webhook (BYOW)"
      },
      "owners": ["devops"]
    }
  ]

this shouldn't trigger any alerts since it also has a filter on "owners"

You can also visit sourcegraph.test:9090/alertmanager to validate the generated config manually